After upgrading python 3.8.6 to 3.9.10 using homebrew, my Cython extensions no longer without explicitly adding /usr/local/include (for my Intel MacBook) or /opt/homebrew/include to the include_dirs of my extension.
My setup.py.in:
import os, sys
from numpy.distutils.core import setup, Extension
from Cython.Build import cythonize
link_arguments = []
extra_include_dirs = []
if (sys.platform == 'darwin'):
link_arguments.append("-Wl,-rpath")
link_arguments.append("-Wl,#loader_path/")
if os.path.exists('/opt/homebrew/'):
extra_include_dirs.append("/opt/homebrew/include/")
else:
extra_include_dirs.append("/usr/local/include/")
else:
link_arguments.append("-Wl,-rpath=${CMAKE_SOURCE_DIR}/lib/")
pynwp_extension = Extension(
name="pynwp",
sources=["${CMAKE_CURRENT_SOURCE_DIR}/lambert.f90", "${CMAKE_CURRENT_SOURCE_DIR}/pynwp.f90", "${CMAKE_CURRENT_SOURCE_DIR}/readAtmosphereGen.f90", "${CMAKE_CURRENT_SOURCE_DIR}/ptogrot.f", "${CMAKE_CURRENT_SOURCE_DIR}/bilin1.f", "${CMAKE_CURRENT_SOURCE_DIR}/fl2pres_f.f","${CMAKE_CURRENT_SOURCE_DIR}/message.c","${CMAKE_CURRENT_SOURCE_DIR}/gridWindDirCorrection.F"],
libraries=["HirlamUtils_fPIC", "eccodes_f90", "jasper"],
library_dirs=["${PROJECT_BINARY_DIR}", "${CMAKE_SOURCE_DIR}/build${CMAKE_BUILD_TYPE}/src/libHirlamUtils/", "/opt/homebrew/lib/"],
extra_link_args = link_arguments,
include_dirs=["${CMAKE_SOURCE_DIR}/include", "/usr/lib64/gfortran/modules/",
"${CMAKE_SOURCE_DIR}/build${CMAKE_BUILD_TYPE}"] + extra_include_dirs,
extra_f90_compile_args=["-DLINUX", "-DIS_LITTLE_ENDIAN", "-DUSEWALLTIME", "-DHAS_BLAS", "-DHAS_LAPACK", "-DGRIB32", "-DTIMING", "-DPREC32", "-fno-whole-file", "-g", "-fbounds-check"]
#compiler_directives={'language_level' : "3"}
)
setup(name="pynwp",
author="me",
author_email="me!me.com",
version="1.0.1",
description="Python wrapper for pynwp",
package_dir={"": "${CMAKE_CURRENT_SOURCE_DIR}"},
url="http://emaddc.eu",
license="MIT License",
ext_modules=[pynwp_extension]
)
In the file above, I have hardcoded the location (temporarily) of the homebrew library dir and added some functionality for the include dir based on /opt/homebrew being found. If I remove this from the file, compilation fails as eccodes.mod cannot be found, see the output when I run the command generate by python/CMake manually:
buildDebug git:(master) ✗ /opt/homebrew/bin/gfortran -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops -I<project_dir>//include -I/usr/lib64/gfortran/modules/ -I<project_dir>//buildDebug -Ibuild/src.macosx-12-arm64-3.9/build/src.macosx-12-arm64-3.9 -I/opt/homebrew/lib/python3.9/site-packages/numpy/core/include -Ibuild/src.macosx-12-arm64-3.9/numpy/distutils/include -I/opt/homebrew/opt/python#3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c -c <project_dir>//src/pynwp/readAtmosphereGen.f90 -o build/temp.macosx-12-arm64-3.9<project_dir>//src/pynwp/readAtmosphereGen.o -DLINUX -DIS_LITTLE_ENDIAN -DUSEWALLTIME -DHAS_BLAS -DHAS_LAPACK -DGRIB32 -DTIMING -DPREC32 -fno-whole-file -g -fbounds-check
f951: Warning: Nonexistent include directory '/usr/lib64/gfortran/modules/' [-Wmissing-include-dirs]
f951: Warning: Nonexistent include directory 'build/src.macosx-12-arm64-3.9/build/src.macosx-12-arm64-3.9' [-Wmissing-include-dirs]
f951: Warning: Nonexistent include directory 'build/src.macosx-12-arm64-3.9/numpy/distutils/include' [-Wmissing-include-dirs]
<project_dir>//src/pynwp/readAtmosphereGen.f90:3:7:
3 | use eccodes
| 1
Fatal Error: Cannot open module file 'eccodes.mod' for reading at (1): No such file or directory
compilation terminated.
This extension is part of a larger project that is build using CMake. When running the gfortran command manually, the same errors indicating that eccodes.mod cannot be found. The file is however located on a default location on the gfortran/gcc search path:
locate eccodes.mod
/opt/homebrew/Cellar/eccodes/2.24.2/include/eccodes.mod
/opt/homebrew/include/eccodes.mod
And the search path for gfortran:
gfortran -E -Wp,-v -
#include <...> search starts here:
/opt/homebrew/include
/opt/homebrew/Cellar/gcc/11.2.0_3/bin/../lib/gcc/11/gcc/aarch64-apple-darwin21/11/include
/opt/homebrew/Cellar/gcc/11.2.0_3/bin/../lib/gcc/11/gcc/aarch64-apple-darwin21/11/include-fixed
/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include
/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/System/Library/Frameworks
End of search list.
Another project uses gcc/ld and has a similar problem. I need to explicitly add LINK_DIRECTORIES(/opt/homebrew/lib) to CMakeList.txt in order for gcc to find the eccodes library. Without that, I get:
gcc-11: warning: this compiler does not support X86 (arch flags ignored)
ld: library not found for -leccodes
collect2: error: ld returned 1 exit status
make[2]: *** [src/smoothModeS-v51/smoothModeS-v51.x] Error 1
make[1]: *** [src/smoothModeS-v51/CMakeFiles/smoothModeS-v51.x.dir/all] Error 2
make: *** [all] Error 2
Adding the paths to CPATH and LIBRARY_PATH had no effect.
This method works but seems hard to maintain. What am I missing and what has changed that gfortran/gcc no longer search in the default paths for but libraries and headers/modules?
EDIT
Just found that the standalone executable that uses similar code as the python extension and also uses eccodes has a similar issue. If I do not include INCLUDE_DIRECTORIES(/opt/homebrew/include) in CMakeLists.txt , I get:
cd <project_dir>/buildDebug/src/collocEHS && /opt/homebrew/bin/gfortran -I<project_dir>/buildDebug -I<project_dir>/include -I<project_dir>/src/readASTERIX2 -I<project_dir>/src/geomag70_linux -I<project_dir>/src/libDTG -fallow-argument-mismatch -ffpe-trap=invalid,zero,overflow -DPREC32 -DLINUX -DIS_LITTLE_ENDIAN -DUSEWALLTIME -DHAS_BLAS -DHAS_LAPACK -DGRIB32 -DTIMING -ffixed-line-length-none -g -fcheck=all -Wall -fcheck=bounds -O0 -g -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.1.sdk -c <project_dir>/src/pynwp/readAtmosphereGen.f90 -o CMakeFiles/collocEHSv2.dir/__/pynwp/readAtmosphereGen.f90.o
<project_dir>/src/pynwp/readAtmosphereGen.f90:3:7:
3 | use eccodes
| 1
Fatal Error: Cannot open module file 'eccodes.mod' for reading at (1): No such file or directory
compilation terminated.
make[2]: *** [src/collocEHS/CMakeFiles/collocEHSv2.dir/__/pynwp/readAtmosphereGen.f90.o] Error 1
make[1]: *** [src/collocEHS/CMakeFiles/collocEHSv2.dir/all] Error 2
make: *** [all] Error 2
So I guess the problem is unrelated to python but more with gcc/gfortran (gcc version 11.2.0 (Homebrew GCC 11.2.0_3).
EDIT 2
A reboot of the laptop fixed the issue with the python extension build and setup.py requiring additional directories. For normal builds using CMake, I still require the extra INCLUDE_DIRECTORIES and LINK_DIRECTORIES order for gcc/gfortran to find libraries installed by brew in /opt/homebrew (or /usr/local for intel MacBook).
As per homebrew devs, this is desired behaviour: /opt/homebrew and /usr/local are "special" directories to be manually added in e.g., CMake projects. This is explained in my bug report on home-brew's GitHub, see https://github.com/Homebrew/homebrew-core/issues/95561.
I haven't been able to confirm this with documentation.
When I am compiling with J8, sometime I am getting the following error:
C:\msys64\usr\bin\make.exe -C firmware all -j8 -O
Compiling file: project/src/tests.c with optimization level: -Os
error: unable to open output file 'build/tests.o': 'no such file or directory'
1 error generated.
This happens randomly.
If I change J8 to J1, then compilation is always successful. There is no problem with path or directory permission as compilation is working with J1 and sometime with J8.
My setup details:
MSYS2
ARM Compiler v6.16
Windows 10
Not sure whether it is a problem of MSYS2 or ARM Compiler or Window specific problem.
I've just installed CLion and i needed compiler. I installed Cygwin.
But there was an error:
CMake Error: Could not find CMAKE_ROOT !!!
CMake has most likely not been installed correctly.
Modules directory not found in
CMake Error: Error executing cmake::LoadCache(). Aborting.
Error code: 1
So, CMake was "bundled" and i tried to install it from site. But there was the next error:
Selected CMake might be incompatible with the Cygwin environment.
In order to run on Cygwin, CMake needs to be specially compiled.
Please select either CMake bundled with CLion or CMake from the Cygwin installation.
CMake Error: The source directory "/cygdrive/c/Users/Пользователь/AppData/Local/Temp/cmake_check_environment" does not exist.
Specify --help for usage, or press the help button on the CMake GUI.
Error code: 1
Then i've tried to install MinGW - this time it was the following problem:
-- The C compiler identification is GNU 8.1.0
-- The CXX compiler identification is GNU 8.1.0
-- Check for working C compiler: D:/MinGW/mingw64/bin/gcc.exe
-- Check for working C compiler: D:/MinGW/mingw64/bin/gcc.exe -- broken
CMake Error at D:/CLion 182.2574.4/bin/cmake/share/cmake-3.11/Modules/CMakeTestCCompiler.cmake:52 (message):
The C compiler
"D:/MinGW/mingw64/bin/gcc.exe"
is not able to compile a simple test program.
It fails with the following output:
Change Dir: C:/Users/Пользователь/AppData/Local/Temp/cmake_check_environment/_build/CMakeFiles/CMakeTmp
Run Build Command:"D:/MinGW/mingw64/bin/mingw32-make.exe" "cmTC_a4ce5/fast"
D:/MinGW/mingw64/bin/mingw32-make.exe -f CMakeFiles\cmTC_a4ce5.dir\build.make CMakeFiles/cmTC_a4ce5.dir/build
mingw32-make.exe[1]: Entering directory 'C:/Users/╧юы№чютрЄхы№/AppData/Local/Temp/cmake_check_environment/_build/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_a4ce5.dir/testCCompiler.c.obj
D:\MinGW\mingw64\bin\gcc.exe -o CMakeFiles\cmTC_a4ce5.dir\testCCompiler.c.obj -c C:\Users\╨Я╨╛╨╗╤М╨╖╨╛╨▓╨░╤В╨╡╨╗╤М\AppData\Local\Temp\cmake_check_environment\_build\CMakeFiles\CMakeTmp\testCCompiler.c
gcc.exe: error: C:\Users\╨Я╨╛╨╗╤М╨╖╨╛╨▓╨░╤В╨╡╨╗╤М\AppData\Local\Temp\cmake_check_environment\_build\CMakeFiles\CMakeTmp\testCCompiler.c: No such file or directory
gcc.exe: fatal error: no input files
compilation terminated.
mingw32-make.exe[1]: *** [CMakeFiles\cmTC_a4ce5.dir\build.make:65: CMakeFiles/cmTC_a4ce5.dir/testCCompiler.c.obj] Error 1
mingw32-make.exe[1]: Leaving directory 'C:/Users/╧юы№чютрЄхы№/AppData/Local/Temp/cmake_check_environment/_build/CMakeFiles/CMakeTmp'
mingw32-make.exe: *** [Makefile:125: cmTC_a4ce5/fast] Error 2
CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
CMakeLists.txt:1 (project)
-- Configuring incomplete, errors occurred!
See also "C:/Users/Пользователь/AppData/Local/Temp/cmake_check_environment/_build/CMakeFiles/CMakeOutput.log".
See also "C:/Users/Пользователь/AppData/Local/Temp/cmake_check_environment/_build/CMakeFiles/CMakeError.log".
Error code: 1
In both cases it's problems with paths, i think. I tried to give root to CLion, gcc, etc. - nothing.
I've tried to find logs or just way to this path - but all was worthless - path doesn't exist.I just can't find out, why that's happening.
Help me, please
Looks like there is a problem with encodings somewhere between windows/cygwin/clion/cmake (I assume you user name is non-ascii).
As a quick workaround try:
installing CLion in the location without non-latin symbols
changing CLion's system and config folders to the location without non-latin symbols. instructions
And to troubleshoot the problem, could you please check if the cygwin's console shows the user home path correctly? (e.g. use ls /cygdrive/c/Users/, then cd to the user home path).
If Cygwin can correctly work with the user home, could you please file a bug report and attach CLion's log files? (Help | Show Log in Explorer)
I'm triing to compile the superoptimizer on windows. (https://github.com/bonzini/superopt)
Unfortunatly my knowledge of make is very limited. I tried:
make CPU=-D386 superopt
Which gives me the error:
cc superopt.c -o superopt
process_begin: CreateProcess(NULL, cc superopt.c -o superopt, ...) failed.
make (e=2): The system cannot find the file specified.
make: *** [superopt] Fehler 2
After setting the compiler to gcc manually with:
make CPU=-D386 superopt CC=gcc
I get the following error:
gcc superopt.c -o superopt
In file included from superopt.c:27:0:
superopt.h:104:2: error: #error You have to choose target CPU type (--with-arch).
#error You have to choose target CPU type (--with-arch).
^
In file included from superopt.h:130:0,
from superopt.c:27:
longlong.h:1465:14: error: unknown type name 'UQItype'
extern const UQItype __clz_tab[];
^
superopt.c:32:21: fatal error: version.h: No such file or directory
compilation terminated.
make: *** [superopt] Fehler 1
It seems it does't properly selects i386.
Any hints would be greatly appreciated.
That is not the source code of
GNU superopt. It is the source code of someone's project
to patch GNU superopt, last updated 2008, and seemingly
abandoned a hard-hat area.
This is the source code of
GNU superopt. Extract the tarball and build as you have attempted
with:
make CC=gcc CPU=-DI386 superopt
Note: I386, not 386.
You will see warnings like:
warning: incompatible implicit declaration of built-in function 'foo'
because the 20 yearold C code does not diligently include the standard
headers that prototype the standard functions that it calls, but superopt will build successfully.
I have trouble in compiling a simple hello world application with CLion and MinGW. The buttons for Run and Debug are grayd out.
I have installed MinGW with mingw32-gcc-g++ and mingw32-make packages.
After that I have install CLion which detected automatically the right folder for MinGW. However make, c compiler and c++ compiler will not get the checkmark.
Also in CMakeError.log file found the following:
Compiling the C compiler identification source file "CMakeCCompilerId.c" failed.
Compiler: C:/MinGW/bin/gcc.exe
Build flags:
Id flags:
The output was:
1
Compiling the C compiler identification source file "CMakeCCompilerId.c" failed.
Compiler: C:/MinGW/bin/gcc.exe
Build flags:
Id flags: -c
The output was:
1
Compiling the C compiler identification source file "CMakeCCompilerId.c" failed.
Compiler: C:/MinGW/bin/gcc.exe
Build flags:
Id flags: -Aa
The output was:
1
Checking whether the C compiler is IAR using "" did not match "IAR .+ Compiler":
gcc.exe: fatal error: no input files
compilation terminated.
Compiling the CXX compiler identification source file "CMakeCXXCompilerId.cpp" failed.
Compiler: C:/MinGW/bin/g++.exe
Build flags:
Id flags:
The output was:
1
Compiling the CXX compiler identification source file "CMakeCXXCompilerId.cpp" failed.
Compiler: C:/MinGW/bin/g++.exe
Build flags:
Id flags: -c
The output was:
1
Checking whether the CXX compiler is IAR using "" did not match "IAR .+ Compiler":
g++.exe: fatal error: no input files
compilation terminated.
Determining if the C compiler works failed with the following output:
Change Dir: D:/Users/Tau/.clion10/system/cmake/generated/94853116/94853116/__default__0/CMakeFiles/CMakeTmp
Run Build Command:C:/MinGW/bin/mingw32-make.exe "cmTryCompileExec3424130643/fast"
C:/MinGW/bin/mingw32-make.exe -f CMakeFiles\cmTryCompileExec3424130643.dir\build.make CMakeFiles/cmTryCompileExec3424130643.dir/build
mingw32-make.exe[1]: Entering directory 'D:/Users/Tau/.clion10/system/cmake/generated/94853116/94853116/__default__0/CMakeFiles/CMakeTmp'
"C:\Program Files (x86)\JetBrains\CLion 138.2344.17\bin\cmake\bin\cmake.exe" -E cmake_progress_report D:\Users\Tau\.clion10\system\cmake\generated\94853116\94853116\__default__0\CMakeFiles\CMakeTmp\CMakeFiles 1
Building C object CMakeFiles/cmTryCompileExec3424130643.dir/testCCompiler.c.obj
C:\MinGW\bin\gcc.exe -o CMakeFiles\cmTryCompileExec3424130643.dir\testCCompiler.c.obj -c D:\Users\Tau\.clion10\system\cmake\generated\94853116\94853116\__default__0\CMakeFiles\CMakeTmp\testCCompiler.c
CMakeFiles\cmTryCompileExec3424130643.dir\build.make:56: recipe for target 'CMakeFiles/cmTryCompileExec3424130643.dir/testCCompiler.c.obj' failed
mingw32-make.exe[1]: Leaving directory 'D:/Users/Tau/.clion10/system/cmake/generated/94853116/94853116/__default__0/CMakeFiles/CMakeTmp'
mingw32-make.exe[1]: *** [CMakeFiles/cmTryCompileExec3424130643.dir/testCCompiler.c.obj] Error 1
Makefile:113: recipe for target 'cmTryCompileExec3424130643/fast' failed
mingw32-make.exe: *** [cmTryCompileExec3424130643/fast] Error 2
I am unable to figure out. I have tried also with cygwin x64 and the same issue occurs. What is wrong?
After so much time spent without any clue, I found that actually the crashing softwares were cc1.exe and cc1plus.exe. After using Dependency Waker tool which led mo to manually adding the zlib1.dll file near these two crashing exe, I found that the real reason was that package mingw32-libz was not installed.
I have installed mingw32-libz package in MinGW and removed manually added dlls and I get all checkmarks in settings.