I can customized the installing process in develop mode by setup.py like below:
from setuptools import setup
from setuptools.command import develop
class CustomDevelop(develop.develop, object):
"""
Class needed for "pip install -e ."
"""
def run(self):
super(CustomDevelop, self).run()
print("CustomDevelop=====================")
install_requires = ["numpy"]
setup(
name="example_package",
version="0.0.1",
packages=["example_package"],
cmdclass={'develop': CustomDevelop},
)
When running python3 -m pip install -vvv -e ., the output is
Running setup.py develop for example-package
Running command python setup.py develop
running develop
running egg_info
writing example_package.egg-info/PKG-INFO
writing dependency_links to example_package.egg-info/dependency_links.txt
writing top-level names to example_package.egg-info/top_level.txt
reading manifest file 'example_package.egg-info/SOURCES.txt'
writing manifest file 'example_package.egg-info/SOURCES.txt'
running build_ext
Creating /home/example_package/venv_example_package/lib/python3.8/site-packages/example-package.egg-link (link to .)
Adding example-package 0.0.1 to easy-install.pth file
Installed /home/lai/example_package
CustomDevelop=====================
However, it's not possible to do it with PEP518 which introduces pyproject.toml. Is there a workaround for this issue?
Related
I have configured a conda environment and created python project. I am using Jenkins freestyle job for project testing. Activated a conda env from Jenkins shell and getting modules not found error, already installed all the required modules to this env.
For information testing cases are working properly when executing pytest server/ command from terminal workspace.
Jenkins Shell Command
#!/bin/bash
source /home/user/anaconda3/etc/profile.d/conda.sh
conda activate Unit-Test
pip list
pytest server/
Using pip list command I try to check the modules installed in env from terminal and Jenkins shell and found some of the libraries are missing when checked from Jenkins Shell.
Already checked with deleting the Jenkins Job caching and created new job to check, but issue still persists. Also tried to install the missing libraries again and again but still having the same issue.
Following libraries are missing in Jenkins shell: (Difference)
asttokens
backcall
debugpy
decorator
executing
ipykernel
pickleshare
psutil
pure-eval
Pygments
python-dateutil
I am not using these libraries directly but boto3 uses python-dateutil as support and I am getting the error:
import boto3
from boto3.session import Session
import botocore.session
import botocore.client
from botocore import waiter, xform_name
from botocore.docs.docstring import WaiterDocstring
from botocore.docs.service import ServiceDocumenter
from botocore.docs.bcdoc.restdoc import DocumentStructure
from botocore.compat import OrderedDict
from dateutil.tz import tzlocal
ModuleNotFoundError: No module named 'dateutil'
I did not get any reason for the issue.
Need Help, Please suggest
Why not install all the modules as part of the job run?
pip install -r requirements.txt
Define all requirements in requirements.txt.
When creating a Python package and uploading it to pypi, it will automatically install the requirements that are put in the setup.py file under install_requires, e.g.
from distutils.core import setup
setup(
name = 'a_package',
packages = ['a_package'],
install_requires=['another_package']
)
When the package has a cython extension (and .pyx files instead of .c/.cpp files), the setup.py file will need to import cython to create an installable extension, e.g.
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
name = 'a_package',
packages = ['a_package'],
install_requires=['another_package'],
cmdclass = {'build_ext': build_ext},
ext_modules = [Extension('the_extension', sources=['a_file.pyx'])]
)
But since Cython is imported before executing the setup part, when trying to install this package through pip from source (rather than from a wheel) downloaded from pypi, it will fail to install due to not being able to import cython, as it has not reached the part with the requirements yet.
I’m wondering what can be done to ensure that a pip install of this package from pypi will install cython before it tries to import it. Adding a requirements.txt with cython does not seem to add automatic-install requirements for files downloaded from pypi.
Now, I realize it’s possible to just pip install cython before pip install thispackage, but I’m wondering if there’s a better fix that would allow to install the package along with cython directly from pypi when it’s not possible to run an additional command (without resorting to uploading the .c. files and ajusting the setup.py file to use them instead of the .pyx).
What you're describing is a "build time dependency", and this is precisely the use case "PEP 518 -- Specifying Minimum Build System Requirements for Python Projects" was created for.
You can specify cython as a build-time dependency by adding a pyproject.toml file like:
[build-system]
requires = ["cython"]
Then when installing your package with a modern version of pip (or another PEP 518 compatible installer), cython will be installed into the build environment before your setup.py script is run.
How do I make pip recompile Cython extensions when I have only changed setup.py when installing in editable mode. Currently it always skips the extensions. There are many questions related to this for distutils but I can't see any answers for pip.
To be clear, I have a setup.py like the following
from distutils.core import setup, Extension
from Cython.Distutils import build_ext
import numpy
ext_modules = []
# simulate_fast
ext_modules += [
Extension("adio.simulating.simulate_fast_c",
sources=["./adio/simulating/simulate_fast/simulate_fast_c.pyx",
"./adio/simulating/simulate_fast/c/simulate_fast.c",
"./adio/simulating/simulate_fast/c/matrix.c"],
include_dirs=[numpy.get_include()],
extra_compile_args=["-Ofast", "-ffast-math", "-march=native"],
language='c',
libraries=["gsl", "openblas"],
define_macros=[('FLOAT32', 1)]
)
]
setup(
name="adio",
packages=["adio"],
cmdclass={'build_ext': build_ext},
ext_modules=ext_modules
)
Now if I delete the ('FLOAT32', 1) in define_macros of course I would like the Cython extension to recompile.
However when I run
python3 -m pip install --editable -U . -v
I receive the following output as part of the output
running develop
running egg_info
writing adio.egg-info/PKG-INFO
writing dependency_links to adio.egg-info/dependency_links.txt
writing top-level names to adio.egg-info/top_level.txt
reading manifest file 'adio.egg-info/SOURCES.txt'
writing manifest file 'adio.egg-info/SOURCES.txt'
running build_ext
skipping './adio/simulating/simulate_fast/simulate_fast_c.c' Cython extension (up-to-date)
I have tried the -I and --force-reinstall flag with pip but it always skips the Cython extension. If I am not using editable mode then I can run
python3 -m pip install -U . -v
and this does recompile. How can I achieve the same thing when using the --editable flag.
Related
distutils ignores changes to setup.py when building an extension?
I'm using python 3.6
Trying to make and exe from my .py file. but when I run python setup.py build
it give me an error
raise ImportError("No module named %r" % name)
ImportError: No module named 'idna'
My setup file is like this
setup file
Save:
# thanks to https://www.youtube.com/watch?v=GSoOwSqTSrs
from cx_Freeze import setup, Executable
setup(
name='KutsalAklinNerde?',
version='0.1', #Further information about its version
description='Parse stuff', #It's description
executables= [Executable("Example.py")])
as setup.py in the same directory of your Example.py (the .py you want to convert to .exe)
And then run >python setup.py build in that directory with command prompt.
I am trying to use python-ldap with AWS Lambda. I downloaded the tarball from : https://pypi.python.org/pypi/python-ldap
and code to use lambda (lambda_function.py)
from ldap_dir.ldap_query.Lib import ldap
and uploaded the zip to Lambda.
where my directory structure is
ldap_dir -> ldap_query -> Lib -> ldap folder
ldap_dir -> lambda_function.py
Am I missing out something?
python-ldap is built on top of native OpenLDAP libraries. This article - even though unrelated to the python ldap module - describes how to bundle Python packages that have native dependencies.
The outline of this is the following:
Create an Amazon EC2 instance with Amazon Linux
Install compiler packages as well as the OpenLDAP developer package. yum install -y gcc openldap-devel
Create a virtual environment: virtualenv env
Activate the virtual environment: env/bin/activate
Upgrade pip (I am not sure this is necessary, but I got a warning without this): pip install --upgrade pip
Install python-ldap: pip install python-ldap
Create a handler Python script, for example, lambda.py with the following code:
import os
import subprocess
libdir = os.path.join(os.getcwd(), 'local', 'lib')
def handler(event, context):
command = 'LD_LIBRARY_PATH={} python ldap.py'.format(libdir)
subprocess.call(command, shell=True)
Implement your LDAP function, in this example ldap.py:
import ldap
print ldap.PORT
Create a zip package, let's say ldap.zip:
zip -9 ~/ldap.zip ldap.py
zip -9 ~/ldap.zip lambda.py
cd env/lib/python2.7/site-packages
zip -r9 ~/ldap.zip *
cd ../../../lib64/python2.7/site-packages
zip -r9 ~/ldap.zip *
Download the zip to your system (or put it into an S3 bucket). Now you can create your Lambda function using lambda.handler as the function name and use the zip file as the code.
I hope this helps.
one more step/check to the solution above:
still you might get No module named '_ldap', then check if the python version that you install on local/EC2 are the same as the Runtime on lambda