I'm not sure how to debug (presumably with GDB) the Python code behind a SWIG interface.
I can use ipdb to watch the execution of Tensorflow's Python code all the way to the SWIG wrapper (e.g. tf_session.TF_Run in session.py), but I would like to debug the C++ code behind the SWIG interface.
Presumably I build Tensorflow with bazel build --config debug, but how do I attach gdb to the resulting code when called from the Python interface?
TensorFlow's C++ code executes in the same process as the Python code that calls it (or, if you are using the distributed version, in the same process as one of the Python programs that created a tf.GrpcServer).
The simplest interface between Python and C++ is the pure-C API in tensor_c_api.h. To intercept one of these calls, you can attach gdb to the process ID of the Python interpreter that is running TensorFlow, and create a breakpoint on one of these functions.
For example, using an interactive Python session, in the first terminal enter:
$ python
>>> import tensorflow
>>> import os
>>> os.getpid()
14680
Then, in another terminal, start gdb:
$ gdb -p 14680
[...]
(gdb) break TF_NewSession
Breakpoint 1 at 0x7f15f450a4d0
(gdb) continue
Continuing.
Back in the Python interpreter, create a new session:
>>> sess = tf.Session()
The interpreter will pause, and your debugger will print something like the following:
Breakpoint 1, 0x00007f15f450a4d0 in TF_NewSession () from [...]/tensorflow/python/_pywrap_tensorflow.so
(gdb) backtrace
#0 0x00007f15f450a4d0 in TF_NewSession () from [...]/tensorflow/python/_pywrap_tensorflow.so
#1 0x00007f15f3ac5cdb in _wrap_TF_NewSession () from [...]/tensorflow/python/_pywrap_tensorflow.so
#2 0x000000000049968d in PyEval_EvalFrameEx ()
#3 0x00000000004a090c in PyEval_EvalCodeEx ()
#4 0x0000000000499a52 in PyEval_EvalFrameEx ()
[...]
You can now use the full power of gdb to debug TensorFlow.
Related
I am trying to debug my tests using pdb (Python debugger) while running them with bazel.
This is a sample test I have:
class TestMembersResource(TestCase):
def test_get(self):
response = self.client.get('/api/v1/members/')
import ipdb; ipdb.set_trace()
self.assertEqual(response.status_code)
When I try to run it with bazel test ... I get the following output:
Traceback (most recent call last):
File "/root/.cache/bazel/_bazel_root/ae988d93859d448ae36776fcb135b36c/execroot/__main__/bazel-out/k8-fastbuild/bin/webserver/members/api/tests/test_members_resource.runfiles/__main__/webserver/members/api/tests/test_members_resource.py", line 22, in test_get
self.assertEqual(response.status_code, 200,
File "/root/.cache/bazel/_bazel_root/ae988d93859d448ae36776fcb135b36c/execroot/__main__/bazel-out/k8-fastbuild/bin/webserver/members/api/tests/test_members_resource.runfiles/__main__/webserver/members/api/tests/test_members_resource.py", line 22, in test_get
self.assertEqual(response.status_code, 200,
File "/usr/lib/python2.7/bdb.py", line 49, in trace_dispatch
return self.dispatch_line(frame)
File "/usr/lib/python2.7/bdb.py", line 68, in dispatch_line
if self.quitting: raise BdbQuit
BdbQuit
Without pdb everything works pretty smooth.
Is there a way to get an interactive shell and use the standard pdb commands with bazel test?
Thanks!
You can do this using the --run_under flag, as mentioned. It's important to note that you need to point to the pdb.py for your python install. To find where to point to, you can do the following:
Check where your python version is installed. This should be using something like python2.7, or python3.6, not just python or python3, as those are frequently just symlinks.
$ which python3.6
/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6
Note that this is where the binary is located, while we want to point to a library file. To do so, replace the last bin with lib, and specify the desired file, something like this:
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/pdb.py
Now you can run your targets like this:
bazel run --run_under="/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/pdb.py"
You need to use --run_under:
bazel test --run_under=/usr/bin/pdb //webserver/members/api/tests:test_members_resource
Alternatively to using bazel's --run_under, you can also just set a breakpoint() (builtin function, no import needed) anywhere is your code and just do a normal bazel run. When the interpreter hits the breakpoint, it will open pdb.
Optional but very helpful: Use pudb via
pip install pudb
export PYTHONBREAKPOINT="pudb.set_trace"
# add breakpoint() to your code
bazel run PYTHON_TARGET
I am debugging a compiled (c++) python module which mysteriously crashes at import time inside docker container. When I run python3 inside gdb, then import, and then try to print backtrace, I only get
# gdb /usr/bin/pyton3
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.
...
(gdb) run -c 'import myModule'
warning: Error disabling address space randomization: Operation not permitted
During startup program terminated with signal SIGSEGV, Segmentation fault.
(gdb) bt
No stack.
(gdb)
Are there some restriction what gdb can(not) do inside a Docker container? The crash in itself is surprising, but the inability to debug it even more so. The module is compiled with -ggdb2 to include as much debugging information as possible.
For the record: debugging (and also strace) need the CAP_PTRACE capability. The problem was solved by running the container with --privileged.
I run python 2.7.13 on windows 7.
I am creating a window with Gtk (from pygobject 3.18.2).I also use win32ui (from pywin32 221).When I import both modules my program runs just fine untill I close it. The window is closing fine, but the python process keeps running and the cmd window that is used to run the script does not return to de cmd prompt.I have to kill python to get back to the promptHere is a simple test script. This does not close proper on my system. If I comment out
#import win32ui
it will close proper
from gi.repository import Gtk
import win32ui
class Window(Gtk.Window):
def __init__(self):
Gtk.Window.__init__(self)
self.connect("destroy", Gtk.main_quit)
self.show_all()
Window()
Gtk.main()
Looks like a conflict between Gtk and win32ui.I need win32gui and win32ui for extracting icons from pe files.What can I do?
On previous versions of PyWin32:
https://sourceforge.net/p/pywin32/bugs/609/
https://sourceforge.net/p/pywin32/bugs/636/
import atexit, os
def taskkill_this():
# kill this process
current_pid = os.getpid()
os.system("taskkill /pid %s /f" % current_pid)
atexit.register(taskkill_this)
Appears to be one provided workaround.
Upon further inspection, it appears to be related to pywin32 and Gtk UI event processes clashing. I would recommend using one or the other rather than both if possible or use a taskkill hack like the one above.
I've also noticed that this has happened on Windows 7 workstations in all cases I have seen.
I have generated a SWIG python binding of a C++ library. I would like for others to be able to script it using python directly, but I would also like to embed python in my application so users can setup python scripts to modify variables at runtime. When I run python or python_d I can import my release and debug versions of my library with no issues and then use them. However, when I try and embed them I can run the following code with no issues in release mode, but in debug mode I get the error below.
Embedded code:
Py_SetProgramName("AnimatLab");
Py_Initialize();
PyRun_SimpleString("import os\n"
"os.chdir(\"C:/Projects/AnimatLabSDK/AnimatLabPublicSource/bin\")\n"
"import AnimatSimPy\n");
Py_Finalize();
Error:
C:\Projects\AnimatLabSDK\AnimatLabPublicSource\bin
Traceback (most recent call last):
File "<string>", line 4, in <module>
ImportError: No module named AnimatSimPy
I know the _AnimatSimPy_d.pyd works in debug mode because I can import and use it when I run python_d, but for some reason it will not run when embedded. Does anyone have any ideas on why this is failing? I am linking the debug version to python27_d.dll on a 64 bit windows 7 machine.
I figured this out. I am not sure why, but in debug mode it required me to append to my paths to add the directory where the dll was and not just set the working directory. For release mode it worked without this. So if I run the code below it works in both debug and release mode from the embedded code.
Py_SetProgramName("AnimatLab");
Py_Initialize();
PyRun_SimpleString("import os\n"
"import sys\n"
"sys.path.append(\"C:/Projects/AnimatLabSDK/AnimatLabPublicSource/bin\")\n"
"os.chdir(\"C:/Projects/AnimatLabSDK/AnimatLabPublicSource/bin\")\n"
"import AnimatSimPy\n");
Py_Finalize();
I'd like to be able to open a Python shell, execute some code defined in a module, then modify the module, then re-run it in the same shell without closing/reopening.
I've tried reimporting the functions/objects after modifying the script, and that doesn't work:
Python 2.7.2 (default, Jun 20 2012, 16:23:33)
[GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from my_module import buggy_function, test_input
>>> buggy_function(test_input)
wrong_value # Returns incorrect result
# Go to the editor, make some changes to the code and save them
# Thought reimporting might get the new version
>>> from my_module import buggy_function, test_input
>>> buggy_function(test_input)
wrong_value # Returns the same incorrect result
Clearly reimporting did not get me the 'new version' of the function.
In this case, it isn't that big a deal to close the interpreter and reopen it. But, if the code I'm testing is complicated enough, sometimes I have to do a fair amount of importing objects and defining dummy variables to make a context that can adequately test the code. It does get annoying to have to do this every time I make a change.
Anyone know how to "refresh" the module code within the Python interpeter?
use imp.reload():
In [1]: import imp
In [2]: print imp.reload.__doc__
reload(module) -> module
Reload the module. The module must have been successfully imported before.