How to debug pip install <package name> - debugging

If I want to debug the setup.py file of my package packagename getting installed via "pip install packagename" , is there a way to do so? I have tried downloading the source, adding set_trace() in setup.py and run:
pip install .
However, as soon I get pdb() prompt, the install fails with error:
processing /Users/skauser/python-ibmdb/IBM_DB/ibm_db
Complete output from command python setup.py egg_info:
> /private/var/folders/b6/pmddncpn77550p8_g9kkx9f40000gp/T/pip-req-build-_fg8s5a2/setup.py(31)<module>()
-> machine_bits = 8 * struct.calcsize("P")
(Pdb)
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/var/folders/b6/pmddncpn77550p8_g9kkx9f40000gp/T/pip-req-build-_fg8s5a2/setup.py", line 31, in <module>
machine_bits = 8 * struct.calcsize("P")
File "/private/var/folders/b6/pmddncpn77550p8_g9kkx9f40000gp/T/pip-req-build-_fg8s5a2/setup.py", line 31, in <module>
machine_bits = 8 * struct.calcsize("P")
File "/Library/anaconda3/lib/python3.7/bdb.py", line 88, in trace_dispatch
return self.dispatch_line(frame)
File "/Library/anaconda3/lib/python3.7/bdb.py", line 113, in dispatch_line
if self.quitting: raise BdbQuit
bdb.BdbQuit
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/b6/pmddncpn77550p8_g9kkx9f40000gp/T/pip-req-build-_fg8s5a2/
Although I can debug the source via: python setup.py build, the behavior that I want to debug is applicable when installed through pip.

Pip is a python program. You can see it's content with cat $(which pip). Then you can copy it to a new file in your project's directory. For example, here is how it looks like for me:
File mypip.py:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import re
import sys
from pip._internal.cli.main import main
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
sys.exit(main())
Then you can add a breakpoint with pdb or use PyCharm as mentioned in the answer from Igor Yudnikov.
For reference, this is how it to looks like in my Pycharm

Oh! Just found a solution, its pretty easy. What is pip? It's a plain python file, which is I think somehow bound to run via python, and under the hood system makes something like
python pip install ...
so, at first we need to know where is pip
which pip
says
/Users/me/PycharmProjects/etlscripts/venv/bin/pip
So, we may go to Pycharm, create config like
then you'll be able to catch your exception. Cheers.

Related

Problem installing importlib on Mac with pip3

Good morning!
Until yesterday I was using python 2.7 and using the following code for writing output files for my scripts:
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
But now I'm upgrading my code to python 3.7, so now I need to use importlib.reloadinstead of just reload. However, typing sudo -H pip3 install importlib on my shell gives me the following error:
Collecting importlib
Using cached https://files.pythonhosted.org/packages/31/77/3781f65cafe55480b56914def99022a5d2965a4bb269655c89ef2f1de3cd/importlib-1.0.4.zip
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/local/lib/python3.7/site-packages/setuptools/__init__.py", line 6, in <module>
import distutils.core
File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/core.py", line 16, in <module>
from distutils.dist import Distribution
File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/dist.py", line 19, in <module>
from distutils.util import check_environ, strtobool, rfc822_escape
File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/util.py", line 9, in <module>
import importlib.util
ModuleNotFoundError: No module named 'importlib.util'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/tmp/pip-install-3cz_3d6n/importlib/
After looking for solutions I also tried to upgrade setuptools, but that didn't fix my problem.
Many thanks in advance!
importlib at PyPI is an outdated package intended for very old Python versions. For new versions of Python importlib is a module from the standard library, you don't need to install it, it's always available.

Can't create virtualenv using Python 3.2. Whodunnit? Pip or setuptools or Python 2.7?

I can't properly install a Python 3.2 based virtual environment using virtualenvwrapper. I have hit a mystery wall and now I am looking for more options to solve my WHODUNNIT? predicament.
Some background
I first tried to install virtualenvwrapper using pip 1.5.6 from Python 2.7 (system's Python default). When I was trying to uninstall/upgrade my pip somehow I broke it. Now if I try:
$ pip
Traceback (most recent call last):
File "/usr/local/lib/python3.2/dist-packages/distribute-0.6.28-py3.2.egg/pkg_resources.py", line 2816, in <module>
working_set.require(__requires__)
File "/usr/local/lib/python3.2/dist-packages/distribute-0.6.28-py3.2.egg/pkg_resources.py", line 690, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/local/lib/python3.2/dist-packages/distribute-0.6.28-py3.2.egg/pkg_resources.py", line 592, in resolve
raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (pip 7.1.2 (/usr/local/lib/python3.2/dist-packages), Requirement.parse('pip==1.5.6'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/pip", line 5, in <module>
from pkg_resources import load_entry_point
File "/usr/local/lib/python3.2/dist-packages/distribute-0.6.28-py3.2.egg/pkg_resources.py", line 2820, in <module>
parse_requirements(__requires__), Environment()
File "/usr/local/lib/python3.2/dist-packages/distribute-0.6.28-py3.2.egg/pkg_resources.py", line 588, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: pip==1.5.6
However, I managed to install PIP3.2.
$ pip3.2 --version
pip 7.1.2 from /usr/local/lib/python3.2/dist-packages (python 3.2)
So I proceeded to install virtualenvwrapper with no problem:
$ sudo pip3.2 install virtualenvwrapper
This is where Python 3.2 resides
$ whereis python3.2.3
whereis python3.2.3
python3.2: /usr/bin/python3.2 /etc/python3.2 /usr/lib/python3.2 /usr/bin/X11/python3.2 /usr/local/lib/python3.2 /usr/include/python3.2 /opt/Python-3.2/bin/python3.2 /usr/share/man/man1/python3.2.1.gz
This is my virtualenv version:
$ virtualenv --version
14.0.6
Error
Before making a virtualenv I modified my .bashrc
export WORKON_HOME=$HOME/.virtualenvs
export PROJECT_HOME=$HOME/admin/workspace
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
export VIRTUALENVWRAPPER_VIRTUALENV=/usr/local/bin/virtualenv
source /usr/local/bin/virtualenvwrapper.sh
After sourcing it (i.e. source ~/.bashrc) I just can't manage to start a virtualenv.
$ mkvirtualenv --python=/usr/bin/python3.2 venv
Running virtualenv with interpreter /usr/bin/python3.2
New python executable in /home/admin/.virtualenvs/venv/bin/python3.2
Also creating executable in /home/admin/.virtualenvs/venv/bin/python
/usr/local/lib/python3.2/dist-packages/virtualenv-14.0.6-py3.2.egg/virtualenv_support/pip-8.0.2-py2.py3-none-any.whl/pip/_vendor/pkg_resources/__init__.py:87: UserWarning: Support for Python 3.0-3.2 has been dropped. Future versions will fail here.
Traceback (most recent call last):
File "/usr/local/lib/python3.2/dist-packages/virtualenv-14.0.6-py3.2.egg/virtualenv.py", line 2304, in <module>
main()
File "/usr/local/lib/python3.2/dist-packages/virtualenv-14.0.6-py3.2.egg/virtualenv.py", line 705, in main
symlink=options.symlink)
File "/usr/local/lib/python3.2/dist-packages/virtualenv-14.0.6-py3.2.egg/virtualenv.py", line 924, in create_environment
download=download,
File "/usr/local/lib/python3.2/dist-packages/virtualenv-14.0.6-py3.2.egg/virtualenv.py", line 843, in install_wheel
cert_data = pkgutil.get_data("pip._vendor.requests", "cacert.pem")
File "/usr/lib/python3.2/pkgutil.py", line 573, in get_data
loader = get_loader(package)
File "/usr/lib/python3.2/pkgutil.py", line 461, in get_loader
return find_loader(fullname)
File "/usr/lib/python3.2/pkgutil.py", line 471, in find_loader
for importer in iter_importers(fullname):
File "/usr/lib/python3.2/pkgutil.py", line 427, in iter_importers
__import__(pkg)
File "/usr/local/lib/python3.2/dist-packages/virtualenv-14.0.6-py3.2.egg/virtualenv_support/pip-8.0.2-py2.py3-none-any.whl/pip/__init__.py", line 15, in <module>
After the error if I type...
$ workon
$
Funny thing is the venv is created in .virtualenvs
Possible causes
Reading elsewhere in the web (I found just one person having a similar problem) it could be solved using a previous version of virtualenv (I tried that to no success), or maybe I'm using an incorrect combination of pip and/or virtualenv and/or virtualenvwrapper. Or perhaps it's my Python2.7 messing around in the background.
UPDATE: Setuptools
Or what about setuptools? Apparently it has to do. I was getting this error when trying with PIP 1.5.6
UserWarning: Support for Python 3.0-3.2 has been dropped. Future versions will fail here.
warnings.warn(msg)
ANOTHER UPDATE: Fixed the original pip
I managed to fix original pip using:
$ wget https://bootstrap.pypa.io/get-pip.py
$ sudo python get-pip.py
$ pip --version
pip 8.0.3 from /usr/local/lib/python2.7/dist-packages (python 2.7)
UPDATE: Virtualwrapper isn't!
In my own 'Whodunnit' saga I'd say virtualwrapper has nothing to do. Here:
$ virtualenv -p /usr/bin/python3.2 ~/.virtualenvs/venv32
Running virtualenv with interpreter /usr/bin/python3.2
New python executable in /home/admin/.virtualenvs/venv32/bin/python3.2
Also creating executable in /home/admin/.virtualenvs/venv32/bin/python
/usr/local/lib/python2.7/dist-packages/virtualenv_support/pip-8.0.2-py2.py3-none-any.whl/pip/_vendor/pkg_resources/__init__.py:87: UserWarning: Support for Python 3.0-3.2 has been dropped. Future versions will fail here.
But problem still persists. What else could I try?
Thanks in advance!
You must use pip < 8 and virtualenv < 14 if you want to use Python 3.2. See also https://virtualenv.pypa.io/en/latest/changes.html#id7
Why do you need to use ancient Python 3.2? The best thing you can do would be to install and use the far superior Python >= 3.4 instead.

Python3.4.1 LINUX - from pip import main raises ImportError: Cannot import name 'main'

I'm using a server for which I am not root, and I have several scripts requiring Python 3.4.1. I built the software from the source tarball using the following syntax
$ ./configure --prefix=$HOME
$ make
$ make install
The software installed successfully, as I was able to launch python3 after modifying my .bash_profile to look in $HOME/bin. However, Pip was not so lucky.
$ pip3.4
Traceback (most recent call last):
File "/home/scnorton/bin/pip3.4", line 7, in <module>
from pip import main
ImportError: cannot import name 'main'
When I tried to run python3 get-pip.py, I get the following error:
Traceback (most recent call last):
File "get-pip.py", line 17474, in <module>
main()
File "get-pip.py", line 17466, in main
bootstrap(tmpdir=tmpdir)
File "get-pip.py", line 17406, in bootstrap
import pip
zipimport.ZipImportError: can't decompress data; zlib not available
I have checked the /usr/lib64 directory and concluded that zlib is installed on the server. I even executed the command module load zlib-1.2.8-1 and tried reinstalling pip again, with no luck. Could the problem be that Python3.4.1 is installed as a 32-bit application? If so, where can I go to get 64-bit Python3.4.1 for Linux?

Using cx_freeze with PyQT5 and Python 3 on MacOSX

I'm trying to use cx_freeze 4.3.3 on a MacOS running 10.9.2 on a very simple PyQt5 script with Python 3.3.
No errors are returned and the .app is output. However when running the .app from terminal I obtain the error:
LSOpenURLsWithRole() failed with error -10810
which according to Apple's documentation is "Unknown Error".
The very simple code I try to run (PyQt5app.py) is:
import sys
from PyQt5.QtWidgets import QApplication, QDialog
app = QApplication(sys.argv)
form = QDialog()
form.show()
app.exec_()
The file setup.py is:
import sys
from cx_Freeze import setup, Executable
base = None
if sys.platform == 'win32':
base = 'Win32GUI'
options = {
'build_exe': {
'excludes': ['Tkinter'] # Sometimes a little finetuning is needed
}
}
executables = [
Executable('PyQt5app.py', base=base)
]
setup(name='PyQt5app',
version='0.1',
description='Sample PyQt5 GUI',
executables=executables,
options=options
)
and when running I call:
sudo python cx_freeze bdist_mac
obtaining this log : http://pastebin.com/VBxyyBRn
with app returning error above.
So, reading around I see it might be a problem related to including qt files in the app (or at least this was the issue on PyQt4) so I try specifying the qt-menu-nib directory:
sudo python setup.py bdist_mac --qt-menu-nib=/Users/franco/Qt5.2.1/5.2.1/clang_64/plugins/platforms/
obtaining this log: http://pastebin.com/TpRdrSmT
and the same not working error.
If I run the app from PyQt5app.app/Contents/MacOS/PyQt5app I get a lot of bootstrap errors:
Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/cx_Freeze-4.3.3-py3.3-macosx-10.9-x86_64.egg/cx_Freeze/initscripts/Console.py", line 27, in <module>
exec(code, m.__dict__)
File "PyQt5app.py", line 5, in <module>
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/importlib/_bootstrap.py", line 1565, in _find_and_load
return _find_and_load_unlocked(name, import_)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/importlib/_bootstrap.py", line 1532, in _find_and_load_unlocked
loader.load_module(name)
File "ExtensionLoader_PyQt5_QtWidgets.py", line 22, in <module>
File "ExtensionLoader_PyQt5_QtWidgets.py", line 14, in __bootstrap__
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/importlib/_bootstrap.py", line 1565, in _find_and_load
return _find_and_load_unlocked(name, import_)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/importlib/_bootstrap.py", line 1532, in _find_and_load_unlocked
loader.load_module(name)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/importlib/_bootstrap.py", line 584, in _check_name_wrapper
return method(self, name, *args, **kwargs)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/importlib/_bootstrap.py", line 495, in set_package_wrapper
module = fxn(*args, **kwargs)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/importlib/_bootstrap.py", line 508, in set_loader_wrapper
module = fxn(self, *args, **kwargs)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/importlib/_bootstrap.py", line 1132, in load_module
fullname, self.path)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/importlib/_bootstrap.py", line 313, in _call_with_frames_removed
return f(*args, **kwds)
SystemError: initialization of sip raised unreported exception
Needless to say the script works fine when launching from terminal:
python PyQt5app.py
I most certainly am doing something wrong, so please, can anybody help me?
So, after long fighting here is the issue: libzmq .
I installed libzmq and specified the --qt-menu-nib option and the simple example above runs in both:
sudo python setup.py build
and
sudo python setup.py bdist_mac
Step by step instructions:
I used mac ports for most of my python33 packages so I sticked to it for the rest.
Libzmq is not available on macports but its dependencies are.
So:
1) install libtool, autoconf, automake:
sudo port install libtool
sudo port install autoconf
sudo port install automake
2) grab the latest version of libzmq from https://github.com/zeromq/libzmq ( I downloaded the ZIP for sake of order ) and unzip/navigate to the folder
/libzmq-master
now the instructions provided in the INSTALL document in the folder are pretty clear, if you installed all the dependencies then you will be fine.
run:
sudo ./autogen.sh
sudo ./configure
sudo make
sudo make install
3) download the latest cx_freeze from https://bitbucket.org/anthony_tuininga/cx_freeze/downloads then unzip/untar navigate to folder and run:
sudo python setup.py build
sudo python setup.py install
now when compiling code for MacOSX that uses Python3.3 and PyQt5 you can run:
sudo python setup.py build
then navigate in the build folder and run the program as:
./nameoftheprogram
once you made sure this work then build the app or dmg (as you prefer) with:
sudo python setup.py bdist_mac --qt-menu-nib=/Users/username/Qt5.2.1/5.2.1/clang_64/plugins/platforms/
where the path is the path to your Qt5 installation. If I don't use the --qt-menu-nib option the app crashes on startup whereas the build works fine.
Hope this will help someone in the future.
I got the same error and recompiling with the latest version of ZMQ (4.0.4) did not solve the issue.
However, by looking in the messages of the /Applications/Utilities/Console.app program, I could see that an extra command line option was given to my Python program. This command line option always started with -psn_0_ followed by a number. Apparently psn stands for Process Serial Number (see: http://hintsforums.macworld.com/showthread.php?t=11978).
My program uses the argparse package to parse the command line arguments and exited with an error because it didn't recognized the -psn option. I now filter it out before parsing with argparse and this solved my problem.

HTTPLIB2 Error installing python-twitter on mac osx

Hello I am trying to install python-twitter and one of the dependencies HTTPLIB2 is giving me a lot of problems. I have searched the internet up and down but maybe I do not know what the right key words are. Here is the problem.
I am installing python-twitter. I install the requirements either by "pip install -r requirements.txt" or doing them one by one through for example "pip install httplib2". Then I run "python setup.py install". It's happy but then "python setup.py test" fails as followed with an error on import HTTPLIB2.
python setup.py test
running test
running egg_info
writing requirements to python_twitter.egg-info/requires.txt
writing python_twitter.egg-info/PKG-INFO
writing top-level names to python_twitter.egg-info/top_level.txt
writing dependency_links to python_twitter.egg-info/dependency_links.txt
reading manifest file 'python_twitter.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching '.DS_Store'
writing manifest file 'python_twitter.egg-info/SOURCES.txt'
running build_ext
Traceback (most recent call last):
File "setup.py", line 73, in <module>
Main()
File "setup.py", line 66, in Main
setuptools.setup(**METADATA)
File
....
<deleted these parts to save space>
module = __import__('.'.join(parts_copy))
File "/Users/ME/Downloads/python-twitter-1.0/twitter_test.py", line 30, in <module>
import twitter
File "/Users/ME/Downloads/python-twitter-1.0/twitter.py", line 65, in <module>
import oauth2 as oauth
File "/usr/local/lib/python2.7/site-packages/oauth2/__init__.py", line 32, in <module>
import httplib2
File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 347
print('%s:' % h, end=' ', file=self._fp)
^
SyntaxError: invalid syntax
But I thought the print error was an incompatibility between python 2 and 3. Why am I getting this while I clearly have python 2.7 (I checked and uninstalled and reinstalled HTTPLIB2).
Thanks much for your help
i had a same issue. It seems to be python2.7 and 3.x version conflict issue and resolved by
python2.7 -m pip install -t lib/ -r requirements.txt
Just in case, this happens when you install httplib2 with a python3 version of pip and then execute it with python2.
This happens for example when creating manually an environment with:
pip install -t lib/ -r requirements
As it happens when creating an AppEngine standard environment vendor folder.
This happens because contrary to most of the libraries, httplib2 has completely different versions for python2 and python3.

Resources