importerror on pip install after pypi install - pip

ref: https://github.com/louking/loutilities/tree/0.14.7
I am getting an importerror after pip install of this package. When I use easy_install of the egg all is well. Any help on how to debug would be appreciated.
Used following commands to install this package to pypi, and pip install using "pip 18.1 from c:\users\lking\anaconda2\lib\site-packages\pip (python 2.7)"
python setup.py install
python setup.py sdist bdist_wheel
twine upload dist/loutilities-0.14.7.*
:
# later
pip install loutilities
When I run the project, see the following
NoAppException: While importing "run", an ImportError was raised:
Traceback (most recent call last):
File "C:\Users\lking\Documents\Lou's Software\projects\contracts\contracts\venv\lib\site-packages\flask\cli.py", line 235, in locate_app
__import__(module_name)
File "C:\Users\lking\Documents\Lou's Software\projects\contracts\contracts\run.py", line 31, in <module>
app = create_app(Development(configpath), configpath)
File "C:\Users\lking\Documents\Lou's Software\projects\contracts\contracts\contracts\__init__.py", line 82, in create_app
from contracts.views.frontend import bp as frontend
File "C:\Users\lking\Documents\Lou's Software\projects\contracts\contracts\contracts\views\frontend\__init__.py", line 18, in <module>
import frontend
File "C:\Users\lking\Documents\Lou's Software\projects\contracts\contracts\contracts\views\frontend\frontend.py", line 21, in <module>
from loutilities.flask_helpers.blueprints import add_url_rules
ImportError: No module named flask_helpers.blueprints
find_packages seems to be finding the appropriate packages:
from setuptools import find_packages
find_packages()
Out[4]: ['loutilities', 'tests', 'loutilities.flask_helpers']
setup.py:
#!/usr/bin/python
# [irrelevant comments deleted]
import glob
import pdb
# home grown
from loutilities import version
from setuptools import setup, find_packages
def globit(dir, filelist):
outfiles = []
for file in filelist:
filepath = '{0}/{1}'.format(dir,file)
gfilepath = glob.glob(filepath)
for i in range(len(gfilepath)):
f = gfilepath[i][len(dir)+1:]
gfilepath[i] = '{0}/{1}'.format(dir,f) # if on windows, need to replace backslash with frontslash
outfiles += [gfilepath[i]]
return (dir, outfiles)
setup(
name = "loutilities",
version = version.__version__,
packages = find_packages(),
# include_package_data = True,
scripts = [
'loutilities/agegrade.py',
'loutilities/apikey.py',
'loutilities/applytemplate.py',
'loutilities/filtercsv.py',
'loutilities/makerst.py',
],
# Project uses reStructuredText, so ensure that the docutils get
# installed or upgraded on the target machine
install_requires = [
'unicodecsv>=0.13.0',
],
# If any package contains any of these file types, include them:
data_files = ([
globit('loutilities', ['*.conf','*.pyc','*.pyd','*.dll','*.h','*.xlsx']),
globit('doc/source', ['*.txt', '*.rst', '*.html', '*.css', '*.js', '*.png', '*.py', ]),
globit('doc/build/html', ['*.txt', '*.rst', '*.html', '*.css', '*.js', '*.png', ]),
globit('doc/build/html/_sources', ['*.txt', '*.rst', '*.html', '*.css', '*.js', '*.png', ]),
globit('doc/build/html/_static', ['*.txt', '*.rst', '*.html', '*.css', '*.js', '*.png', ]),
globit('doc/build/html/_images', ['*.png', ]),
]),
entry_points = {
'console_scripts': [
'agegrade = loutilities.agegrade:main',
'apikey = loutilities.apikey:main',
'applytemplate = loutilities.applytemplate:main',
'filtercsv = loutilities.filtercsv:main',
'makerst = loutilities.makerst:main',
],
},
zip_safe = False,
# metadata for upload to PyPI
description = 'some hopefully useful utilities',
long_description=open("README.md").read(),
license = 'Apache License, Version 2.0',
author = 'Lou King',
author_email = 'lking#pobox.com',
url = 'http://github.com/louking/loutilities',
# could also include long_description, download_url, classifiers, etc.
)

After pip install loutilities I see an unknown directory loutilities at the top-level of my virtual environment. The directory contains a few *.pyc but not any subdirectory. I suspect import loutilities.flask_helpers tries to find a subdirectory loutilities/flask_helpers and fails.
I think the top-level directory is from the code
globit('loutilities', ['*.conf','*.pyc','*.pyd','*.dll','*.h','*.xlsx']),
I suspect there is a bug in globit() that puts the data in the wrong directory.

Related

How to solve ValueError('Invalid async_mode specified') for flask-socketio?

I'm testing a flask-socketio server in bitbucket pipeline. It failed with the following messages:
Traceback (most recent call last):
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.7/site-packages/flask_failsafe.py", line 29, in wrapper
return func(*args, **kwargs)
File "/opt/atlassian/pipelines/agent/build/main.py", line 89, in create_app
return cell_data_api.create_app()
File "/opt/atlassian/pipelines/agent/build/cell_data_api/__init__.py", line 30, in create_app
socketio.init_app(app)
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.7/site-packages/flask_socketio/__init__.py", line 243, in init_app
self.server = socketio.Server(**self.server_options)
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.7/site-packages/socketio/server.py", line 127, in __init__
self.eio = self._engineio_server_class()(**engineio_options)
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.7/site-packages/engineio/server.py", line 145, in __init__
raise ValueError('Invalid async_mode specified')
ValueError: Invalid async_mode specified
Traceback (most recent call last):
File "/.pyenv/versions/3.7.4/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/.pyenv/versions/3.7.4/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/opt/atlassian/pipelines/agent/build/main.py", line 117, in <module>
socketio.run(APP, host='0.0.0.0', port=8080, debug=True, use_reloader=True)
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.7/site-packages/flask_socketio/__init__.py", line 564, in run
if app.debug and self.server.eio.async_mode != 'threading':
AttributeError: 'NoneType' object has no attribute 'eio'
My main.py file looks like:
import os
from cell_data_api import socketio
# Detect if we are running in App Engine
# Make sure this does NOT start if we are running a Cloud Function
if os.getenv('APP_ENGINE', '') == 'TRUE':
import cell_data_api
APP = cell_data_api.create_app()
if __name__ == '__main__':
from flask_failsafe import failsafe
#failsafe
def create_app():
# note that the import is *inside* this function so that we can catch
# errors that happen at import time
import cell_data_api
# If `entrypoint` is not defined in app.yaml, App Engine will look for an app
# called `app` in `main.py`.
return cell_data_api.create_app()
APP = create_app()
# This is used when running locally only. When deploying to Google App
# Engine, a webserver process such as Gunicorn will serve the app. This
# can be configured by adding an `entrypoint` to app.yaml.
socketio.run(APP, host='0.0.0.0', port=8080, debug=True, use_reloader=True)
Main.py imports from cell_data_api.py, which looks like:
import os
from flask import Flask
from flask_cors import CORS
# import eventlet
from engineio.async_drivers import eventlet
from flask_socketio import SocketIO
socketio = SocketIO(
always_connect=True,
logger=True,
async_mode=eventlet,
cookie=...,
ping_timeout=...
)
def create_app():
# create and configure the app
app = Flask(__name__)
CORS(app)
......
socketio.init_app(app)
# ensure the instance folder exists
try:
os.makedirs(app.instance_path)
except OSError:
pass
return app
My environment is Python 3.7 with installation packages:
[dev-packages]
alembic = "*"
flask_failsafe = "*"
wcwidth = "*"
[packages]
flask = "*"
absl-py = "*"
flask-cors = "*"
grpcio = "*"
transitions = "*"
sqlalchemy-json = "*"
sqlalchemy = "1.3.0"
flask_socketio='*'
eventlet='*'
Unlike the other two questions I found about the same error, I'm not using pyinstaller or cx_Freeze.
The async_mode parameter takes a string as an argument.
Instead of this:
async_mode=eventlet,
Do this:
async_mode='eventlet',

poetry install | SolverProblemError Because my_project depends on string (*) which doesn't match any versions, version solving failed

I am yet to use poetry to run project, so excuse lack of understanding.
I successfully installed the poetry python library manager, using:
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python3
Next step poetry install initially returned this error:
me#LAPTOP-G1DAPU88:~/.ssh/workers-python/workers$ poetry install
RuntimeError
Poetry could not find a pyproject.toml file in /home/me/.ssh/workers-python/workers or its parents
at ~/.poetry/lib/poetry/_vendor/py3.8/poetry/core/factory.py:369 in locate
365│ if poetry_file.exists():
366│ return poetry_file
367│
368│ else:
→ 369│ raise RuntimeError(
370│ "Poetry could not find a pyproject.toml file in {} or its parents".format(
371│ cwd
372│ )
373│ )
I soon realised I needed my own made pyproject.toml file. Running poetry install again yielded:
$ poetry install
TOMLError
Invalid TOML file /home/me/.ssh/workers-python/workers/pyproject.toml: Key "json " already exists.
at ~/.poetry/lib/poetry/_vendor/py3.8/poetry/core/toml/file.py:34 in read
30│ def read(self): # type: () -> "TOMLDocument"
31│ try:
32│ return super(TOMLFile, self).read()
33│ except (ValueError, TOMLKitError) as e:
→ 34│ raise TOMLError("Invalid TOML file {}: {}".format(self.path.as_posix(), e))
35│
36│ def __getattr__(self, item): # type: (str) -> Any
37│ return getattr(self.__path, item)
38│
Above error indicates there were duplicate entries.
Running poetry install again with the now updated pyproject.toml file in cwd threw this error (in the post's title):
$ poetry install
Creating virtualenv my_project-1_EUeV5I-py3.8 in /home/me/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies... (28.4s)
SolverProblemError
Because my_project depends on string (*) which doesn't match any versions, version solving failed.
at ~/.poetry/lib/poetry/puzzle/solver.py:241 in _solve
237│ packages = result.packages
238│ except OverrideNeeded as e:
239│ return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
240│ except SolveFailure as e:
→ 241│ raise SolverProblemError(e)
242│
243│ results = dict(
244│ depth_first_search(
245│ PackageNode(self._package, packages), aggregate_package_nodes
However, temporarily removing all instances = "*" gave me this error of \n on line 12... which doesn't appear to be there:
$ poetry install
TOMLError
Invalid TOML file /home/me/.ssh/workers-python/workers/pyproject.toml: Unexpected character: '\n' at line 12 col 5
at ~/.poetry/lib/poetry/_vendor/py3.8/poetry/core/toml/file.py:34 in read
30│ def read(self): # type: () -> "TOMLDocument"
31│ try:
32│ return super(TOMLFile, self).read()
33│ except (ValueError, TOMLKitError) as e:
→ 34│ raise TOMLError("Invalid TOML file {}: {}".format(self.path.as_posix(), e))
35│
36│ def __getattr__(self, item): # type: (str) -> Any
37│ return getattr(self.__path, item)
38│
me#LAPTOP-G1DAPU88:~/.ssh/workers-python/workers$ cat pyproject.toml
[tool.poetry]
name = "my_project"
version = "0.1.0"
description = "Top-level package for my_project."
authors = [""]
packages = [
{ include = "my_project"},
]
[tool.poetry.dependencies]
python = "^3.8"
click # Suspect line
I have reverted this.
Current pyproject.toml:
[tool.poetry]
name = "data_simulator"
version = "0.1.0"
description = "Top-level package for data_simulator."
authors = ["iotahoe <iotahoe#iotahoe.com>"] # daniel.bell#hitachivantara.com / daniel#iotahoe.com
packages = [
{ include = "data_simulator"},
]
[tool.poetry.dependencies]
python = "^3.8"
click = "*"
#logging = "*"
#os = "*"
#pathlib = "*"
#time = "*"
numpy = "*"
pandas = "*"
#json = "*"
#random = "*"
faker = "*"
transformers = "4.4.2"
#re = "*"
#itertools = "*"
#datetime = "*"
#requests = "*"
#copy = "*"
#collections = "*"
#collections.abc = "*"
#multiprocessing = "*"
#multiprocessing.dummy = "*"
nltk = "*"
#nltk.corpus = "*"
#string = "*"
[tool.poetry.dev-dependencies]
isort = "5.6.4"
black = "^20.8b1"
invoke = "^1.4.1"
coveralls = "^2.2.0"
pytest = "^3.0"
flake8 = "^3.8.3"
mypy = "^0.782"
[[tool.poetry.source]]
name = "azure"
url = "https://pkgs.dev.azure.com/iotahoe/Halo/_packaging/private-sources/pypi/simple/"
secondary = true
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
Note: 'name', 'authors', 'include', 'url' have been censored.
As a general advise I recommend to use poetry's command line instead of creating/manipulating the pyproject.toml.
Start with a poetry init or poetry init -n and add your dependencies with poetry add.
The problem with your current pyproject.toml is, that you declare built-in packages as dependencies, like os, pathlib, string and others. This is why you receive the message Because my_project depends on string (*) which doesn't match any versions, version solving failed., which means poetry cannot find any matching package information in the repository.
tl;dr: Flush the *.egg-info directories before running poetry lock.
This answer is not strictly related to the current issue, but a similar error message can appear in other circumstances, so I think it's valuable to share it here.
If you are locking in a project where sub-dependencies are directly available on the file system, some *.egg-info directories may interfere with the locking process, causing issues when trying to run poetry install in a context where those *.egg-info files are missing. To avoid the problem: Flush the *.egg-info directories prior to locking. You should then have an updated poetry.lock file with more content.

Why is pypandoc giving the error "RuntimeError: Missing format!" in this setup.py script?

I have a setup.py script for a module. This setup.py script features conversion of a Markdown README file to reStructuredText (suitable for PyPI). When an attempt is made to install this module using pip (sudo pip install supermodule), the following error is presented:
Collecting supermodule
Downloading supermodule-2017.1.12.2329.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-GijMOu/supermodule/setup.py", line 43, in <module>
main()
File "/tmp/pip-build-GijMOu/supermodule/setup.py", line 14, in main
long_description = pypandoc.convert("README.md", "rst"),
File "/usr/local/lib/python2.7/dist-packages/pypandoc/__init__.py", line 50, in convert
outputfile=outputfile, filters=filters)
File "/usr/local/lib/python2.7/dist-packages/pypandoc/__init__.py", line 68, in _convert
raise RuntimeError('Missing format!')
RuntimeError: Missing format!
What is going wrong and how can I fix this?
The setup.py script is as follows:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import os
import pypandoc
import setuptools
def main():
setuptools.setup(
name = "supermodule",
version = "2017.01.12.2329",
description = "super module",
long_description = pypandoc.convert("README.md", "rst"),
url = "https://github.com/user/supermodule",
author = "A. Person",
author_email = "ap#sern.ch",
license = "GPLv3",
py_modules = [
"supermodule"
],
install_requires = [
"opencv",
"docopt",
"propyte",
"shijian",
"tonescale"
],
scripts = [
"supermodule_test.py"
],
entry_points = """
[console_scripts]
supermodule = supermodule:supermodule
"""
)
def read(*paths):
with open(os.path.join(*paths), "r") as filename:
return filename.read()
if __name__ == "__main__":
main()

Windows ImportError numpy.core.multiarray failed to import

I write the application using modules PyQt4, cv2(v.2.4.13), numpy(v.1.11.1) and etc. I use Python (2.7.11 on win32), Windows7 (x64).
Before the compiling (using py2exe) on Windows my application work correctly (run from python).
In setup.py:
...
options = {
'py2exe': {
...
'compressed': True,
'includes': ['cv2', 'numpy', 'sip', 'lxml._elementpath', 'PyQt4.QtCore', 'PyQt4.QtGui' ],
'dll_excludes': ['MSVCP90.dll', 'w9xpopen.exe', 'Qwt.pyd', 'tcl85.dll', 'tk85.dll']
}
}
...
After compiling on Windows and trying to start the apps (*.exe file) the system return this error:
File "cv2.pyc", line 12, in module
File "cv2.pyc", line 10, in __load
ImportError: numpy.core.multiarray failed to import.
The exception raised in line:
import cv2
In the folder named 'dist' there is file numpy.core.multiarray.pyd.
In library.zip there are file cv2, folder numpy.
SOLVED
The problem was in setup.py in the py2exe section.
Also I copied numpy-atlas.dll from C:\Python27\Lib\site-packages\numpy\core to site-packages for succesful build.
setup.py:
excludes = [
'Tkconstants', 'Tkinter', 'tcl', '_ssl', 'bz2',
'_testcapi', 'pyexpat', 'select'
]
options = {
'py2exe': {
'compressed': True,
'excludes': excludes,
'includes': ['sip', 'lxml._elementpath', 'PyQt4.QtCore', 'PyQt4.QtGui', 'cv2', 'numpy'],
'dll_excludes': ['MSVCP90.dll', 'w9xpopen.exe', 'Qwt.pyd', 'tcl85.dll', 'tk85.dll', 'MSVCR90.DLL']
}
}
I had a similar problem for me the solution was as easy as moving the image file to the folder where the executable was created. This worked for both py2exe and pyinstaller.

cx_freeze: How do I add package files into library.zip?

I've noticed that pytz misses zoneinfo folder when I try to roll a zip for Windows. Right now I have a workaround that I use after python setup.py build, namely
7z a -xr!*.py* build\exe.win32-2.7\library.zip C:\Python27\Lib\site-packages\pytz
Is there a proper way to achieve that from setup.py or something?
You could fix this, adding the following method:
def include_files():
path_base = "C:\\Python27\\Lib\\site-packages\\pytz\\zoneinfo\\"
skip_count = len(path_base)
zip_includes = [(path_base, "pytz/zoneinfo/")]
for root, sub_folders, files in os.walk(path_base):
for file_in_root in files:
zip_includes.append(
("{}".format(os.path.join(root, file_in_root)),
"{}".format(os.path.join("pytz/zoneinfo", root[skip_count:], file_in_root))
)
)
return zip_includes
Then, into setup.py file:
build_exe_options = {"packages": ["os"],
"excludes": ["tkinter"],
"zip_includes": include_files(),
...
}
Hope that helps
I've solved this problem in Python 3.4 in the following way
import pytz
setup(
...
options = {'build_exe':
{'include_files': (pytz.__path__[0],), ...},
},
)
Then pytz is included unzipped with all its time zones

Resources