How to use debug libraries on ubuntu - debugging

Question

My current problem is with libwebkitgtk-3.0-0, but I guess this problem is generic enough.
My application is crashing somewhere in the webkit code. My assumption is we are doing something stupid and want to find out what. Easiest thing will be to set a breakpoint or use debug version of library.
How do I get exact source code with which library was built? I am getting stack trace after it dumps the core, but line number gdb is saying do not match with those I see in the code. In other words if I install libwebkitgtk-3.0-0 I want to get exact source code of that.
I have installed debug version of webkit library. Do these debug versions have same functionality as if you are compiling webkit with with --enable-debug flag? Debug versions of webkit enable logging based on WEBKIT_DEBUG environment variable, but I could not get the same logging even if use debug version of library.
How to use the debug version I managed to compile? I managed to compile webkit on my machine and tried fiddling with load paths and such. My application does not pickup new shared library no matter what I do - I can tell based on user agent signature. At one point, I managed to pick up the library, but then SSL stops working. Same SSL problem does happen with GtkLauncher. So I am making mistake somewhere.
Thanks for the pointers.

Answer 1

TL;DR: Install libwebkitgtk-3.0-0-dbg , then you have the necessary debug symbols.
##For debug symbols, you don't usually have to install from source.
As you know, to get debug symbols for software you're building yourself, you can run GCC with -g.
For software installed through your operating system's package manager (which includes libwebkitgtk-3.0-0, here), at least for official packages, there are usually also packages providing debug symbols.
You don't actually need to have a debug build of a program or library to get a symbolic stack trace in gdb. gdb also supports files providing "add-on" debug symbols in /usr/lib/debug.
You use Ubuntu, according to the tags on your question. On Ubuntu, debug symbol packages are available in two varieties: -dbg and -dbgsym. A program or library located at /path gets debug symbols at /usr/lib/debug/path.
##-dbg Packages
These packages are often named differently from the corresponding packages providing the actual executables or library files. They are often named similarly to -dev packages (which provide header files) and -doc packages. A -dbg package sometimes has less library version numbering in the name than the actual library packages, sometimes covering binaries provided in multiple other packages.
For example, libgtkmm-3.0-1's corresponding -dbg package is libgtkmm-3.0-dbg.
On the other hand, sometimes a -dbg package is named the same as the package whose symbols it provides (except the -dbg suffix). For example, libwebkitgtk-3.0-0's corresponding -dbg package is libwebkitgtk-3.0-0-dbg. That's the one you want.
You can install it in the Software Center or by running:
sudo apt-get update && sudo apt-get install libwebkitgtk-3.0-0-dbg
Now, when you debug a program that links to a library provided by libwebkitgtk-3.0-0, gdb will automatically load symbols from a file provided by libwebkitgtk-3.0-0-dbg.
##-dbgsym Packages
Sometimes binary executables provided by an official package don't have symbols provided in any -dbg package. When this happens, usually you can install the -dbgsym package.
Unlike -dbg packages, -dbgsym packages:
are almost always simply (and predictably) named X-dbgsym where X is the package providing the program or library itself.
are provided by special software sources (repositories), not the same software sources as provide the corresponding program/library packages and -dbg packages.
Since -dbgsym packages are in separate repositories, you must enable these repositories. Their DEB lines are:
deb http://ddebs.ubuntu.com YOUR_RELEASE main restricted universe multiverse
deb http://ddebs.ubuntu.com YOUR_RELEASE-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com YOUR_RELEASE-security main restricted universe multiverse
deb http://ddebs.ubuntu.com YOUR_RELEASE-proposed main restricted universe multiverse
To enable them, you can run these commands (adapted from DebuggingProgramCrash by "Contributors to the Ubuntu documentation wiki", section 2):
echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-security main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse
" | sudo tee -a /etc/apt/sources.list.d/ddebs.list
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 428D7C01
sudo apt-get update
Leave out the italicized lines, if you are on a development release (alpha or beta). Make sure to add them if you continue using the release once stable, though.
Those commands do three things:
Create the file /etc/apt/sources.list.d/ddebs.list (which contains the DEB lines).
Import the signing key for these repositories.
Update your system's information about what packages and versions are available for installation from where.
So if you ever want to use the -dbgsym-provided symbols instead of the -dbg provided symbols, the -dbgsym package for libwebkitgtk-3.0-0 is (in accordance with the simple naming convention above) libwebkitgtk-3.0-0-dbgsym.
You can have both -dbg and -dbgsym packages installed on the same system, but not if they provide symbols for any of the same files. So libwebkitgtk-3.0-0-dbg and libwebkitgtk-3.0-0-dbgsym conflict with each other; they cannot both be installed (at the same time).
##Using the Symbols
On most Unix-like OSes, the debugger will automatically look for installed symbols. Ubuntu is no different--in Ubuntu, gdb automatically looks for them in /usr/lib/debug. So you don't need to do anything special.
However, if you ever did need to tell gdb to load a specific debug symbol file, you would use the -s file flag. See the GNU manual and gdb(1) for details.

Answer 2

1) When I need to dig into a library that I installed through a package, the first thing I do is install it from source. I mean configur/make/make install. I typically put the source code in /usr/local/src and install it in /usr/local . This, in my opinion is the most reliable way of running the exact code for which you have the source.
3)
How to use the debug version I managed to compile?
This sounds like you did what I described above. What you need to do is make sure that your software is using the include and link directories that are hosting your compiled, debug enabled, library. Meaning making sure that -I/usr/local/include and -L /usr/local/lib flags are set and they come before /usr/include and /usr/lib.
You can be even more certain by removing the binary version of the libraries from the ubuntu installation, making sure that that version that you built and installed is the only version present on the hard disk. This way you will know for sure that you were able to configure your app to use that library. Otherwise it'll just fail, instead of you constantly wondering whether it's using the new library or the old library.
2) Typically yes. But it'd depend on how the library is written, and what the ubuntu packager decided to do.
Once you compile the program using your locally built library, see if you are getting the same exact error first. If not then that is also a data point. Maybe the problem got fixed since the last time ubuntu packaged the library. Maybe the library is not packaged properly and that's the problem. You might even get new errors, because the ubuntu packager configured the library a certain way so that it'd work and you didn't do the same thing. You will get interesting leads anyway.
Good luck

Answer 3

#Eliah's answer tells how to get symbols in a convenient way.
The question remains, "how do I get the exact source code?".
I normally do apt-get source <pkgname> which is well and good except then I must manually tell gdb dir <path-to-wherever-I-put-the-source> and woe betide if it is a package like eglibc where one must figure out that the path references are from the nss subdirectory, not the root.
On RHEL one simply does e.g. yum install --enable-repo rhel-debuginfo libX11-debuginfo (just yum install libX11-debuginfo on CentOS 7) and instantly you get full symbols and source in gdb with no extra messing around. I'm still looking for that convenience on Ubuntu.

Answer 4

In order to debug against dynamic libraries you can add the dgb gears with symbol and source distro packages as suggested.
Then it's necessary to check if the compilation directories of the debug symbol table matches the paths of the installed source, if it's not you should mapping the paths in gdb. Following the commands to enable debugging of glibc
$ objdump -g /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.27.so | sed -n '/<.*>\s\+DW_AT_comp_dir/ {s/\s\+<.*>\s\+//; p;}' | sort | uniq
DW_AT_comp_dir : (indirect string, offset: 0x1127a): /build/glibc-OTsEL5/glibc-2.27/malloc
...
DW_AT_comp_dir : (indirect string, offset: 0xd139): /build/glibc-OTsEL5/glibc-2.27/stdio-common
DW_AT_comp_dir : (indirect string, offset: 0xef40): /build/glibc-OTsEL5/glibc-2.27/libio
$ ls -ld glibc-2.27/{stdio-common,libio}
drwxrwxr-x 3 fusillator fusillator 12288 feb 1 2018 glibc-2.27/libio
drwxrwxr-x 3 fusillator fusillator 4096 feb 1 2018 glibc-2.27/stdio-common
$ gdb ./hello
Reading symbols from ./hello...done.
(gdb) set substitute-path /build/glibc-OTsEL5/glibc-2.27 glibc-2.27
(gdb) b main
Breakpoint 1 at 0x63e: file hello.c, line 10.
(gdb) run
Starting program: hello
Breakpoint 1, main () at hello.c:10
10 printf("hello world\n");
(gdb) s
_IO_puts (str=0x5555555546e4 "hello world") at ioputs.c:33
33 {
(gdb) backtrace
#0 _IO_puts (str=0x5555555546e4 "hello world") at ioputs.c:33
#1 0x000055555555464a in main () at hello.c:10