Can I use Python Debugger In Bazel Test - debugging

I am trying to debug my tests using pdb (Python debugger) while running them with bazel.
This is a sample test I have:
class TestMembersResource(TestCase):
def test_get(self):
response = self.client.get('/api/v1/members/')
import ipdb; ipdb.set_trace()
self.assertEqual(response.status_code)
When I try to run it with bazel test ... I get the following output:
Traceback (most recent call last):
File "/root/.cache/bazel/_bazel_root/ae988d93859d448ae36776fcb135b36c/execroot/__main__/bazel-out/k8-fastbuild/bin/webserver/members/api/tests/test_members_resource.runfiles/__main__/webserver/members/api/tests/test_members_resource.py", line 22, in test_get
self.assertEqual(response.status_code, 200,
File "/root/.cache/bazel/_bazel_root/ae988d93859d448ae36776fcb135b36c/execroot/__main__/bazel-out/k8-fastbuild/bin/webserver/members/api/tests/test_members_resource.runfiles/__main__/webserver/members/api/tests/test_members_resource.py", line 22, in test_get
self.assertEqual(response.status_code, 200,
File "/usr/lib/python2.7/bdb.py", line 49, in trace_dispatch
return self.dispatch_line(frame)
File "/usr/lib/python2.7/bdb.py", line 68, in dispatch_line
if self.quitting: raise BdbQuit
BdbQuit
Without pdb everything works pretty smooth.
Is there a way to get an interactive shell and use the standard pdb commands with bazel test?
Thanks!

You can do this using the --run_under flag, as mentioned. It's important to note that you need to point to the pdb.py for your python install. To find where to point to, you can do the following:
Check where your python version is installed. This should be using something like python2.7, or python3.6, not just python or python3, as those are frequently just symlinks.
$ which python3.6
/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6
Note that this is where the binary is located, while we want to point to a library file. To do so, replace the last bin with lib, and specify the desired file, something like this:
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/pdb.py
Now you can run your targets like this:
bazel run --run_under="/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/pdb.py"

You need to use --run_under:
bazel test --run_under=/usr/bin/pdb //webserver/members/api/tests:test_members_resource

Alternatively to using bazel's --run_under, you can also just set a breakpoint() (builtin function, no import needed) anywhere is your code and just do a normal bazel run. When the interpreter hits the breakpoint, it will open pdb.
Optional but very helpful: Use pudb via
pip install pudb
export PYTHONBREAKPOINT="pudb.set_trace"
# add breakpoint() to your code
bazel run PYTHON_TARGET

Related

cx_Freeze Mac-build stopped at _ctypes for Homebrew-and-Python

We use cx_Freeze to produce standalone binary build of our python application under Mac OS. The build runs well under the build machine (which has Homebrew-and-Python installed), but failed in client machine with following error messages.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/cx_Freeze/initscripts/__startup__.py", line 14, in run
module.run()
File "/usr/local/lib/python2.7/site-packages/cx_Freeze/initscripts/Console.py", line 26, in run
exec(code, m.__dict__)
File "./utest2.py", line 15, in <module>
from ttLib import *
File "ttLib.py", line 848, in init ttLib
import ctypes
File "/usr/local/Cellar/python#2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 7, in <module>
from _ctypes import Union, Structure, Array
ImportError: dlopen(/Users/gff/src/TextSeek_test/build/test_build/lib/_ctypes.so, 2): Symbol not found: __PySlice_AdjustIndices
Referenced from: /Users/gff/src/TextSeek_test/build/test_build/lib/_ctypes.so
Expected in: flat namespace
in /Users/gff/src/TextSeek_test/build/test_build/lib/_ctypes.so
By browsing stackoverflow answers about "__PySlice_AdjustIndices" and "flat namespace", we suspect this error was caused by the python conflict between system-default version and homebrew version.
Then we use "brew install python#2" in client machine, and the ctypes error disappeared. We run "brew uninstall python#2", this error returned back.
The question is : how can we embed necessary part of homebrew-python to bypass this ctype error?
We used "otool -L lib/_ctypes.so" to know the depenency of dynamic-linked file, it only showed out one file "/usr/lib/libSystem.B.dylib". This file seems to have no relation with python.
Any advice to move forward?
Dev Machine Info: Mac High Sierra, python 2.7.15 64bit. Use homebrew-python to build.
setup.py for cx_Freeze :
build_exe_options = {
"packages": ["wcwidth", "watchdog", "xlrd", "jinja2", "subprocess"],
"excludes": [ "AppKit", "Carbon", "CoreFoundation", "Finder", "Foundation", "FSEvents", "objc"],
"include_msvcr": True,
"zip_include_packages":["winshell", "wcwidth", "watchdog", "pyhk", "xlrd", "jinja2",\
"argh", "ctypes", "email", "encodings" ]
}
exeList = [Executable( "utest2.py", base = None, targetName= "utest") ]
setup( name = "XXX",
description = u"XXX desc",
options = {
"build_exe": build_exe_options,
'bdist_mac': {
'bundle_name': "XXX",
}
},
executables = exeList)
I found the solution by myself.
This issue is caused by wrongly linked dylib files. We use "otool -L" to find the dependency and re-link them with "install_name_tool -change" one by one. Finally the program works.
On the client machine, after installation of the frozen application, try to manually copy all dynamic libraries (.so, .dylib, ...?) which are in the lib subdirectory of the build directory into the build directory itself. Copy also all dynamic libraries which are in the build directory into its lib subdirectory. Does this solve the problem? On the build machine, you should of course use the same python version to produce the frozen application than you use to run the unfrozen application.
If 1. solves the problem, find out which dynamic libraries need to be copied in order that the application works (i.e. find out which of the manual copy actions you've done are really necessary). Use the include_files list of the build_exe options in your setup script to let cx_Freeze include the dynamic library automatically at the build step. You can use a tuple (source, destination) as item in the include_files list to let cx_Freeze copy a file from source to a specific destination into the build directory.
Explanation: I don't know Mac/OS. But I believe the issue you describe could be related to a similar issue with Microsoft Visual C++ Redistributable DLLs under Windows: cx_Freeze 5.1.1 includes packages in a lib subdirectory of the build directory, in the previous versions of cx_Freeze they used to be in the build directory itself. Some dynamic libraries need to be found in the build directory but are erroneously included in the lib subdirectory by cx_Freeze or vice versa. On the build machine it is usually not a problem, because a copy of the library is usually found in the system path, but on the client machine often no copy of the library can be found on the system path, or an incompatible one.
By the way the build_exe option "include_msvcr": True does not seem to work in cx_Freeze5.1.1, see this issue.

cannot run scons - unable to find files

So I've installed scons via Anaconda's conda install scons under Windows 10 (Python 3.6) and could not execute it via the command line so I've added C:\Users\D\Anaconda3\envs\py36\Scripts\ to my Path (even tough C:\Users\Dominik\Anaconda3 was already in there).
Now I can execute it in the Powershell, but I get an error, because it is unable find engine files:
scons
SCons import failed. Unable to find engine files in:
C:\Users\D\Anaconda3\envs\py36\Scripts\..\engine
C:\Users\D\Anaconda3\envs\py36\Scripts\scons-local-3.0.1
C:\Users\D\Anaconda3\envs\py36\Scripts\scons-local
C:\Users\D\Anaconda3\scons-3.0.1
C:\Users\D\Anaconda3\Lib\site-packages\scons-3.0.1
C:\Users\D\Anaconda3\scons
C:\Users\D\Anaconda3\Lib\site-packages\scons
Traceback (most recent call last):
File "C:\Users\D\Anaconda3\envs\py36\Scripts\scons.py", line 192, in <module>
import SCons.Script
ImportError: No module named 'SCons'
Does someone know how to further investigate/fix this problem?
I found it thanks to nekomatic.
If you use the anaconda prompt (with the correct environment activated or in my case the correct shortcut leading to anaconda prompt in that environment), scons runs just fine.
See this link explaining how anaconda uses the path on windows and whats the advantage of using the anaconda prompt (mainly it dynamically manages PATH correctly for that session only with full knowledge of conda environments).

cx_freeze builds Mac program that runs from command line but dies when clicked

I have a Python 2.7/PyQt4 program that I am attempting to freeze with cx_freeze. The program also uses requests, serial, xml.etree.ElementTree, and collections. Using the unmodified setup.py generated by cxfreeze-quickstart-2.7, I can successfully build as both a console program (python setup.py build) and an .app (python setup.py bdist_mac) in Yosemite using Python from macports. If I run the program directly either from the app bundle or the dist:
$ build/MacDISE-1.0.app/Contents/MacOS/macdise
$ dist/macdise
It runs exactly as expected. If I open from the command line:
$ open -a /Users/jeffemandel/macdise/build/MacDISE-1.0.app
I get the dreaded
LSOpenURLsWithRole() failed for the application /Users/jeffemandel/macdise/build/MacDISE-1.0.app with error -10810.
I worked through a number of potential issues raised by Dan McCombs (distutils.util.get_platform, sys.arg), but these don’t seem to be the problem. Through brute force trial and error, I found that if I put all of my code in a separate module, simply importing that module (without actually invoking it) causes the 10810 error, so I figured it was finding a library when run from the command line, but not from the app. I put the dist directory on a thumb drive and ran it on another Mac that doesn't have Python, Qt4, etc installed, and got this:
packages/cx_Freeze/initscripts/Console.py", line 27, in <module> File "macdise.py", line 4, in <module>
File "ExtensionLoader_PyQt4_QtGui.py", line 11, in <module>
ImportError: dlopen(/Volumes/NO NAME/dist/PyQt4.QtGui.so, 2): Library not loaded: /opt/local/Library/Frameworks/QtGui.framework/Versions/4/QtGui
Referenced from: /Volumes/NO NAME/dist/PyQt4.QtGui.so
Reason: image not found
When I look in dist, there is a file QtGui that is the same size as the one in /opt. So it seems the failure is dlopen(PyQt4.QtGui.so) returning a hard-coded path to the QtGui library. I'm guessing the solution is simple, but I haven't stumbled across it yet.
Update: I looked at the libraries in build/Contents/MacOS/PyQt4.*.so with otool, and these all have #executable_path (as opposed to hard-coded paths in dist). My stupid. So I repeated the process of moving the program, only using the MacOS folder rather than the dist folder, and executing macdise from the command line on my wife's MBP. The problem turned out to be in the way I was looking for the included_files. I changed this to:
if getattr(sys, 'frozen', False):
uiName = os.path.join(os.path.dirname(sys.executable), "tabDISE.ui" )
else:
uiName = "tabDISE.ui"
and it runs. What would have saved me a day would be a way to automagically dump the error message generated when executing from the command line to the console log. If someone knows how to do this, it would be a big help.

iPython nbconvert not working with headings

Question:
How can I convert ipynb files to PDF if they have headings? I can't make it with the normal command.
Why I ask so:
I've been searching info for many days to get over this problem but I resign and ask it in here. I'm running iPython notebook (with python 3.4) on Windows 8.1 and when I use the command:
ipython nbconvert my file.ipynb --to latex --post PDF
to convert my notebooks to pdf it only works when my file has no headings. If I have the same file and I don't write any heading it works smoothly but the problem comes when I have to put one. It can't even convert the notebook to anything (not HTML, not just LaTeX format...).
My LaTeX distribution is working fine, I can make .tex documents with headings and so on... but the trouble is when I use the nbconvert to make PDF files with headings.
The error it throws is so long so here I paste the first section of it:
[NbConvertApp] Using existing profile dir: 'C:\\Users\\Me\\.ipython\\profile_default'
[NbConvertApp] Converting notebook headers.ipynb to latex
[NbConvertApp] Support files will be in headers_files\
[NbConvertApp] Loaded template article.tplx
Traceback (most recent call last):
File "C:\Program Files (x86)\Anaconda 3\lib\site-packages\IPython\utils\_process_win32.py", line 76, in _find_cmd
from win32api import SearchPath
ImportError: No module named 'win32api'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files (x86)\Anaconda 3\Scripts\ipython-script.py", line 5, in <module> sys.exit(start_ipython())
File "C:\Program Files (x86)\Anaconda 3\lib\site-packages\IPython\__init__.py", line 120, in start_ipython
return launch_new_instance(argv=argv, **kwargs)
...
FINALLY I got it!
First of all, when analysing the Error Code, we can see that the program asks us for the 'win32api'. To solve this problem, one must open the Anaconda Command Prompt as administrator and write:
conda install pywin32
With this, we will fix the missing package. But it doesn't end here, because if we try it again, a longer error will pop up so we also need to install the pandoc pack. In order to do this, we must close the command prompt and download it from here and run it with everything else closed.
Then, one can run smoothly the nbconvert command for everything, yay! :D
So remember, before using nbconvert, be sure that you have:
pywin32
pandoc

Embedded SWIG python debug version will not let me import

I have generated a SWIG python binding of a C++ library. I would like for others to be able to script it using python directly, but I would also like to embed python in my application so users can setup python scripts to modify variables at runtime. When I run python or python_d I can import my release and debug versions of my library with no issues and then use them. However, when I try and embed them I can run the following code with no issues in release mode, but in debug mode I get the error below.
Embedded code:
Py_SetProgramName("AnimatLab");
Py_Initialize();
PyRun_SimpleString("import os\n"
"os.chdir(\"C:/Projects/AnimatLabSDK/AnimatLabPublicSource/bin\")\n"
"import AnimatSimPy\n");
Py_Finalize();
Error:
C:\Projects\AnimatLabSDK\AnimatLabPublicSource\bin
Traceback (most recent call last):
File "<string>", line 4, in <module>
ImportError: No module named AnimatSimPy
I know the _AnimatSimPy_d.pyd works in debug mode because I can import and use it when I run python_d, but for some reason it will not run when embedded. Does anyone have any ideas on why this is failing? I am linking the debug version to python27_d.dll on a 64 bit windows 7 machine.
I figured this out. I am not sure why, but in debug mode it required me to append to my paths to add the directory where the dll was and not just set the working directory. For release mode it worked without this. So if I run the code below it works in both debug and release mode from the embedded code.
Py_SetProgramName("AnimatLab");
Py_Initialize();
PyRun_SimpleString("import os\n"
"import sys\n"
"sys.path.append(\"C:/Projects/AnimatLabSDK/AnimatLabPublicSource/bin\")\n"
"os.chdir(\"C:/Projects/AnimatLabSDK/AnimatLabPublicSource/bin\")\n"
"import AnimatSimPy\n");
Py_Finalize();

Resources