I have a cython file combined.pyx, which merges several pyx files together:
include file1.part.pyx
include file2.part.pyx
...
I also have a setup.py:
from distutils.core import setup
from Cython.Distutils import build_ext, Extension
setup(
ext_modules=[Extension(
"bla.combined",
["src/bla/combined.pyx"])],
requires=['Cython'],
cmdclass={'build_ext': build_ext})
Which I run like this:
python setup.py build_ext --build-lib src
The problem I'm running into is that the setup only looks at combined.pyx when determining whether or not it needs to run again. It doesn't pay attention to file1.part.pyx, so when I modify file1.part.pyx and re-run the setup nothing happens:
python2.7 setup.py build_ext --build-lib src
running build_ext
skipping 'src/bla/combined.c' Cython extension (up-to-date)
Process finished with exit code 0
How do I tell cython/python that it needs to also check file1.part.pyx and file2.part.pyx when determining whether to recompile combined.pyx?
The fix was to cythonize the extension before giving it to setup.
A fixed setup.py:
from distutils.core import setup
from Cython.Distutils import Extension
from Cython.Build import cythonize
setup(
ext_modules=cythonize(Extension(
"bla.combined",
["src/bla/combined.pyx"])),
requires=['Cython'])
Related
I can customized the installing process in develop mode by setup.py like below:
from setuptools import setup
from setuptools.command import develop
class CustomDevelop(develop.develop, object):
"""
Class needed for "pip install -e ."
"""
def run(self):
super(CustomDevelop, self).run()
print("CustomDevelop=====================")
install_requires = ["numpy"]
setup(
name="example_package",
version="0.0.1",
packages=["example_package"],
cmdclass={'develop': CustomDevelop},
)
When running python3 -m pip install -vvv -e ., the output is
Running setup.py develop for example-package
Running command python setup.py develop
running develop
running egg_info
writing example_package.egg-info/PKG-INFO
writing dependency_links to example_package.egg-info/dependency_links.txt
writing top-level names to example_package.egg-info/top_level.txt
reading manifest file 'example_package.egg-info/SOURCES.txt'
writing manifest file 'example_package.egg-info/SOURCES.txt'
running build_ext
Creating /home/example_package/venv_example_package/lib/python3.8/site-packages/example-package.egg-link (link to .)
Adding example-package 0.0.1 to easy-install.pth file
Installed /home/lai/example_package
CustomDevelop=====================
However, it's not possible to do it with PEP518 which introduces pyproject.toml. Is there a workaround for this issue?
I just read the Cython Pure Python Mode documentation and I'm not sure if I understand it right. It sounds as if I could keep all my Python files as they are, add *.pxd files where I declare Cython types. In the setup.py, I still add
from setuptools import setup
from Cython.Build import cythonize
setup(
ext_modules = cythonize(
"A.py",
compiler_directives={'language_level' : "3"}
)
)
When I run python setup.py build_ext --inplace it actually builds the .so file.
What happens when I create the sdist / bdist, upload them to PyPI and a user does not have a matching platform? They will download the sdist, sure. I guess pip / setuptools will automatically try to compile the extension modules (A.py) and I guess if that works, it is fine. But what if cythonize fails? Will it still install the package and use the pure Python code?
I don't think so. I believe a failure in setup.py aborts installation completely.
You can try to declare an extension optional but there're reports that doesn't really work. Could be an issue with older setuptools.
When creating a Python package and uploading it to pypi, it will automatically install the requirements that are put in the setup.py file under install_requires, e.g.
from distutils.core import setup
setup(
name = 'a_package',
packages = ['a_package'],
install_requires=['another_package']
)
When the package has a cython extension (and .pyx files instead of .c/.cpp files), the setup.py file will need to import cython to create an installable extension, e.g.
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
name = 'a_package',
packages = ['a_package'],
install_requires=['another_package'],
cmdclass = {'build_ext': build_ext},
ext_modules = [Extension('the_extension', sources=['a_file.pyx'])]
)
But since Cython is imported before executing the setup part, when trying to install this package through pip from source (rather than from a wheel) downloaded from pypi, it will fail to install due to not being able to import cython, as it has not reached the part with the requirements yet.
I’m wondering what can be done to ensure that a pip install of this package from pypi will install cython before it tries to import it. Adding a requirements.txt with cython does not seem to add automatic-install requirements for files downloaded from pypi.
Now, I realize it’s possible to just pip install cython before pip install thispackage, but I’m wondering if there’s a better fix that would allow to install the package along with cython directly from pypi when it’s not possible to run an additional command (without resorting to uploading the .c. files and ajusting the setup.py file to use them instead of the .pyx).
What you're describing is a "build time dependency", and this is precisely the use case "PEP 518 -- Specifying Minimum Build System Requirements for Python Projects" was created for.
You can specify cython as a build-time dependency by adding a pyproject.toml file like:
[build-system]
requires = ["cython"]
Then when installing your package with a modern version of pip (or another PEP 518 compatible installer), cython will be installed into the build environment before your setup.py script is run.
How do I make pip recompile Cython extensions when I have only changed setup.py when installing in editable mode. Currently it always skips the extensions. There are many questions related to this for distutils but I can't see any answers for pip.
To be clear, I have a setup.py like the following
from distutils.core import setup, Extension
from Cython.Distutils import build_ext
import numpy
ext_modules = []
# simulate_fast
ext_modules += [
Extension("adio.simulating.simulate_fast_c",
sources=["./adio/simulating/simulate_fast/simulate_fast_c.pyx",
"./adio/simulating/simulate_fast/c/simulate_fast.c",
"./adio/simulating/simulate_fast/c/matrix.c"],
include_dirs=[numpy.get_include()],
extra_compile_args=["-Ofast", "-ffast-math", "-march=native"],
language='c',
libraries=["gsl", "openblas"],
define_macros=[('FLOAT32', 1)]
)
]
setup(
name="adio",
packages=["adio"],
cmdclass={'build_ext': build_ext},
ext_modules=ext_modules
)
Now if I delete the ('FLOAT32', 1) in define_macros of course I would like the Cython extension to recompile.
However when I run
python3 -m pip install --editable -U . -v
I receive the following output as part of the output
running develop
running egg_info
writing adio.egg-info/PKG-INFO
writing dependency_links to adio.egg-info/dependency_links.txt
writing top-level names to adio.egg-info/top_level.txt
reading manifest file 'adio.egg-info/SOURCES.txt'
writing manifest file 'adio.egg-info/SOURCES.txt'
running build_ext
skipping './adio/simulating/simulate_fast/simulate_fast_c.c' Cython extension (up-to-date)
I have tried the -I and --force-reinstall flag with pip but it always skips the Cython extension. If I am not using editable mode then I can run
python3 -m pip install -U . -v
and this does recompile. How can I achieve the same thing when using the --editable flag.
Related
distutils ignores changes to setup.py when building an extension?
Using the latest pyinstaller on Windows 7 to make a standalone exe (-F), when running the exe:
ImportError: cannot import name QtGui
In the pyinstaller hooks directory there is special handling for PyQt4 though not PySide.
Hoping for a workaround for this or something to try.
Environment
Windows 7 64-bit
Python 2.7 32-bit
PYTHONHOME=c:\python27
PYTHONPATH=c:\python27\lib
PYTHONLIB=c:\python27\libs\python27.lib;c:\python27\lib\site-packages
Steps
1. Add PySide from http://releases.qt-project.org/pyside/1.1.1/PySide-1.1.1qt474.win32-py2.7.exe
2. Unzip https://github.com/pyinstaller/pyinstaller/zipball/develop to c:\pyinstaller1.5.1
3. Run the commands below against a .py file containing simply:
from PySide import QtGui
[...or QtCore or or.]
Run
c:\pyinstaller1.5.1>pyinstaller.py -F import_test.py
108 INFO: wrote c:\pyinstaller1.5.1\import_test.spec
171 INFO: Testing for ability to set icons, version resources...
296 INFO: ... resource update available
312 INFO: UPX is not available.
4321 INFO: checking Analysis
4382 INFO: checking PYZ
4430 INFO: checking PKG
4446 INFO: building because c:\pyinstaller1.5.1\build\pyi.win32\import_test\import_test.exe.manifest changed
4446 INFO: building PKG out00-PKG.pkg
16782 INFO: checking EXE
16782 INFO: rebuilding out00-EXE.toc because pkg is more recent
16782 INFO: building EXE from out00-EXE.toc
16799 INFO: Appending archive to EXE c:\pyinstaller1.5.1\dist\import_test.exe
c:\pyinstaller1.5.1>dist\import_test.exe
Traceback (most recent call last):
File "<string>", line 23, in <module>
ImportError: cannot import name QtGui
Note
At the end of the PySide install (as admin), this message:
close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr
If that is about post install it can be handled manually:
c:>python.exe c:\Python27\Scripts\pyside_postinstall.py -install
Generating file C:\python27\qt.conf...
PySide installed in c:/python27/Lib/site-packages/PySide...
The PySide extensions were successfully installed.
Workaround. This worked:
# Various imports, whatever, using normal sys.path, for example:
import os, sys, re, time, random
import subprocess, psutil
# Save sys.path
sys_path_saved = sys.path
# Limit sys.path for PySide import
sys.path = ['c:\\python27\\lib\\site-packages']
# PySide imports with limited sys.path
from PySide import QtGui, QtCore
from PySide.QtGui import QApplication, QLineEdit
from PySide.QtCore import QSettings, Qt
# Reset sys.path to original
sys.path = sys_path_saved
# Remainder of code...
Pyinstaller 1.5.1 should do a fine job of locating dependencies, and often does.
However, all of many many attempts to use its pathex or hiddenimports in .spec failed.
Modifying my environment variables also failed.
Extracting various module files manually from from .egg sometimes worked.
However for PySide imports, the sys.path temporary limitation above was the workaround that worked.
Update: Unfortunately the exe only works on a machine with Python/Pyside installed, doesn't work on XP without Python.
For me what worked straight away was to copy/paste the two folders PySide6 and shiboken6 from my python installation folder (PYTHON_FOLDER/Lib/site-packages) to the dist/APP_NAME folder resulting from pyinstaller SCRIPT_NAME.