I've used Ubuntu 20.04.3, and when I run
readelf --string-dump=.gnu_debuglink /usr/bin/cp
I got weird debug filename while expecting something like cp.debug
String dump of section '.gnu_debuglink':
[ 0] 674b3a5e9ca27e34cf3517aa997ba91ce6e0a0.debug
[ 31] k`-
This is the original Ubuntu image, no modification at all.
Is there any reason for this?
I got weird debug filename while expecting something like cp.debug
There is nothing weird about this name.
There are two common schemes for associating the program and debug info for that program -- name of the program, or its linker build-id.
You can find linker build-id with readelf -n /usr/bin/cp. Here is the output on my system:
Displaying notes found in: .note.gnu.build-id
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring) Build ID: 2f6b630344b1b72875f756dce05a40186d18c6d8
Displaying notes found in: .note.ABI-tag
Owner Data size Description
GNU 0x00000010 NT_GNU_ABI_TAG (ABI version tag) OS: Linux, ABI: 3.2.0
Chances are, the linker build-id for your version of cp is 674b3a5e9ca27e34cf3517aa997ba91ce6e0a0.
Using linker build-id is especially convenient for programs which can appear under several different names (where using just the program name you would have to create several copies (or links) of the debug info).
Also, if the program is updated, its linker build-id will change, and so there is no chance that the "stale" debug info would be loaded by the debugger.
Related
I have some Cgo code that I'm linking into my Go binary. I've got Cgo running and building my code and wrapper. After some recent changes, I started getting a double-free in my C++ that I'm linking in. I've tried running my binary under lldb and it does trap the malloc panic, but the symbols are not especially useful.
In vanilla C or C++ I've used -g3 to get rich debugging symbols that includes variable names and source. This makes using lldb much more productive. However, I'm having some issues getting these symbols to show up in my go binary. I've noticed that in the backtrace my function appears as main'foo, where foo is the name of my function. There is no other debug info present though, all I get is a trace of assembly and memory pointers/registers.
I've tried invoking go build with CGO_CFLAGS="-g3" CGO_CXXFLAGS="-g3" but the binary still doesn't have the symbols. I've also tried adding -g3 to the CFLAGS/CXXFLAGS in my .go file where I set other flags (before import "C") but this doesn't seem to work either. I can't think of any other way to get this debugging info added into my binary - is there some Go-specific flag or build sequence that enables this?
I don't know how the process of go & C++ linking works. But a brief description of how debug information is handled on OS X might help you figure out where the debug information is being lost.
On OS X as on most systems, the debug information for an individual source file compile goes into the .o file made from it. You can verify that your .o file got debug information in a variety of ways, here's one:
lldb -o "image dump sections" Target.o --batch | grep DWARF
0x00000300 container [0x0000000000061538-0x0000000000782c77) rwx 0x00061cb8 0x0072173f 0x00000000 Target.o.__DWARF
0x00000009 dwarf-str [0x0000000000061538-0x00000000004eeabc) rwx 0x00061cb8 0x0048d584 0x02000000 Target.o.__DWARF.__debug_str
0x0000000a dwarf-loc [0x00000000004eeabc-0x00000000004ef493) rwx 0x004ef23c 0x000009d7 0x02000000 Target.o.__DWARF.__debug_loc
0x0000000b dwarf-abbrev [0x00000000004ef493-0x00000000004f09a7) rwx 0x004efc13 0x00001514 0x02000000 Target.o.__DWARF.__debug_abbrev
0x0000000c dwarf-info [0x00000000004f09a7-0x00000000006ea7f1) rwx 0x004f1127 0x001f9e4a 0x02000000 Target.o.__DWARF.__debug_info
0x0000000d dwarf-ranges [0x00000000006ea7f1-0x00000000006ec481) rwx 0x006eaf71 0x00001c90 0x02000000 Target.o.__DWARF.__debug_ranges
0x0000000e dwarf-macinfo [0x00000000006ec481-0x00000000006ec482) rwx 0x006ecc01 0x00000001 0x02000000 Target.o.__DWARF.__debug_macinfo
0x0000000f apple-names [0x00000000006ec482-0x000000000071134e) rwx 0x006ecc02 0x00024ecc 0x02000000 Target.o.__DWARF.__apple_names
0x00000010 apple-objc [0x000000000071134e-0x0000000000711372) rwx 0x00711ace 0x00000024 0x02000000 Target.o.__DWARF.__apple_objc
0x00000011 apple-namespaces [0x0000000000711372-0x00000000007116d6) rwx 0x00711af2 0x00000364 0x02000000 Target.o.__DWARF.__apple_namespac
0x00000012 apple-types [0x00000000007116d6-0x0000000000748797) rwx 0x00711e56 0x000370c1 0x02000000 Target.o.__DWARF.__apple_types
0x00000015 dwarf-line [0x000000000075d348-0x0000000000782c77) rwx 0x0075dac8 0x0002592f 0x02000000 Target.o.__DWARF.__debug_line
If you don't see anything here, then the compiler isn't emitting debug information...
The next step is specific to Darwin, instead of putting the debug information into the output of the link stage, the debug info is left in the .o files, and a "debug map" is inserted into the output image. That's how the debugger finds its way back to the .o files. You can see that by doing:
$ nm -ap <YourBinary> | grep OSO
you should see a list of all your .o files here. If you don't then at some point in the build process your binary is getting stripped (using at least strip -S) You have to find out when that is happening and not do that. Also check that the .o files are still where the entries you see from the command above say they are. It may be some part of the build process is moving them around, and the debugger can't find them anymore.
On my fedora box I have installed a lot of separate debug infos.
sudo dnf debuginfo-install <list of packets>
Now, if I debug some simple code it needs very long until some symbol is displayed or some values are printed. It is quite clear that is absolutly needed to evaluate all the installed symbol files to get all information.
But if I have a problem, say on a lib like goocanvas I only want to have my local debug smbols generated with my own compiled code with -g option and the only the debug infos for goocanvas libs.
How can that kind of selection be achieved? Only by renaming the folder of debug info files and generate a copy of needed ones? Maybe as a symlink? Or is there a common selection option anywhere?
You can skip all debug info from shared libraries and only load goocanvas lib symbols. Here is a sample of how to do it in gdb session:
[ ~]$ gdb -q /your/binary
(gdb) set auto-solib-add off
(gdb) start
Temporary breakpoint 1, 0x000055555564edd0 in main ()
(gdb) sharedlibrary goocanvas
From gdb doc:
If your program uses lots of shared libraries with debug info that
takes large amounts of memory, you can decrease the gdb memory
footprint by preventing it from automatically loading the symbols from
shared libraries. To that end, type set auto-solib-add off before
running the inferior, then load each library whose debug symbols you
do need with sharedlibrary regexp, where regexp is a regular
expression that matches the libraries whose symbols you want to be
loaded.
See also this related question: How to prevent GDB from loading debugging symbol for a (large) library?
Setup:
Toolchain: gcc-arm-none-eabi-5_2-2015q4-20151219
Target: STM429i-disco board
I want to run gcov and get real time report generated in target as per below link:
https://mcuoneclipse.com/2014/12/26/code-coverage-for-embedded-target-with-eclipse-gcc-and-gcov/
First, sucessfully have compiled my code (POSIX compliant NUTTX OS) with -fprofile-arcs & -ftest-coverage flags & got generated the .gcno files for my src files.
second, sucesfully have linked with -fprofile-arcs flags enabled and using the libgcov.a file (part of the toolchain) and the final binary is generated.
Now, I dont know what changes are needed in my test application to invoke gcov, generate report & dump report.
Another problem is, gcov functions are with HIDDEN attribute in libgcov.a as below.
9: 00000000 4 FUNC GLOBAL HIDDEN 1 __gcov_flush
9: 00000000 4 FUNC GLOBAL HIDDEN 1 __gcov_init
so, I could not invoke as I need.
Any inputs in getting the .gcda file generated would be of great help.
Can you look for gcov_exit instead? It is similar to __gcov_flush. Typically, it's one of gcov_exit and __gcov_flush that would be there and you can use any.
In case this is not there or is also hidden, you can use this approach which I tried for one of my projects. I picked (and modified for various reasons) the implementation of gcov_exit from gcc source code (of version matching my toolchain) (available at https://github.com/reeteshranjan/libgcov-embedded) and plugged it in my project. With everything else remaining the same (the compiler flags etc.), I was able to then break into gcov_exit and follow the rest of the approach in the blog link you have mentioned.
I have a few Windows 7 machines that I am not able to read their memory dumps. I found something that I suspect may be related, but am not positive:
https://twitter.com/aionescu/status/634028737458114560
I also found this: http://support.microsoft.com/kb/2528507
However, the scenario message regarding wow64exts given in the doc is not seen in any of my dumps. I also cannot apply that hotfix at this time to test it. So I'm just looking for some more information or opinions.
I'm able to open any other OS dump as well as my own system's Windows 7 dump, but there are 2 other machines that run Win 7 and it's telling me I have the wrong kernel symbols.
I have tried clearing out my symbol cache, reinstalled the Windows SDK, and also tried to open the dumps on two other machines with the same result. If it matters, the crash is manually created using the scroll lock method.
Symbol path: SRV*c:\symbols*http://msdl.microsoft.com/download/symbols;
Seeing these errors: followed by "Type referenced: nt!_KPRCB"
Does anyone know about the issue mentioned by Alex in the twitter link and if it's possibly related to what I'm seeing?
Update 2015-10-22:
With the Microsoft patch day (2015-10-13) and KB3088195, symbols are available again.
However, symbols for the broken version have not been provided, so below may still be useful.
Microsoft has already published "good" symbols for ntdll in the past, containing type information like _TEB or _KPRCB. Starting from mid of July 2015, Microsoft has still published symbols for ntdll, but not containing that information.
So it depends on the version of ntdll whether you get type information or not. Old dumps referencing an old version of ntdll will download old PDBs containing type information while new dumps reference new versions of ntdll and WinDbg (or any other debugger) downloads PDBs without type information.
Could Microsoft remove type information of "good" symbols retroactively, thus making them "bad"?
Yes. As described in this answer, there is a tool to remove type information from existing PDBs. Doing that and replacing the PDB would result in such an effect.
Can Microsoft publish the "good" version of those PDBs which are currently "bad"?
That's hard to tell, since we don't know whether Microsoft has kept a copy of the "good" version so they could replace the "bad" version on the symbol server with the "good" one. Rebuilding ntdll from the same source code and thus creating new PDBs sounds possible, but the PDB gets a new time stamp and checksum. This can potentially be corrected manually, especially be Microsoft, since they should have the knowledge about the PDB internal format, but IMHO it's unlikely they'll do that. Things may go wrong and MS will hardly have tests to guarantee the correctness of such a thing.
So what can I do?
IMHO you can do nothing to really correct the situation.
You could assume that the types in ntdll have not changed so much. This would allow you to take an older version of wntdll.pdb and the new version of ntdll.dll and apply ChkMatch -m to it. This will copy the timestamp and checksum from the DLL to the PDB. After you did that (in an empty folder), replace the existing wntdll.pdb in your symbols directory with the hacked one.
WinDbg walkthrough (with output shortened to relevant things). I am using the latest version of wntdll.pdb I could find on my PC.
WARNING: doing the following may fix the type information but will likely destroy the correctness of the callstacks. Since any changes in the implementation (which are likely for security fixes) will change the method offsets.
0:005> dt nt!_PEB
*************************************************************************
*** ***
*** Either you specified an unqualified symbol, or your debugger ***
...
*** Type referenced: nt!_PEB ***
*** ***
*************************************************************************
Symbol nt!_PEB not found.
0:005> lm m ntdll
start end module name
773f0000 77570000 ntdll (pdb symbols) e:\debug\symbols\wntdll.pdb\FA9C48F9C11D4E0894B8970DECD92C972\wntdll.pdb
0:005> .shell cmd /c copy C:\Windows\SysWOW64\ntdll.dll e:\debug\temp\ntdllhack\ntdll.dll
1 file(s) copied.
0:005> .shell cmd /c copy "E:\Windows SDk\8.0\Debuggers\x86\sym\wntdll.pdb\B081677DFC724CC4AC53992627BEEA242\wntdll.pdb" e:\debug\temp\ntdllhack\wntdll.pdb
1 file(s) copied.
0:005> .shell cmd /c E:\debug\temp\ntdllhack\chkmatch.exe -m E:\debug\temp\ntdllhack\ntdll.dll E:\debug\temp\ntdllhack\wntdll.pdb
...
Executable: E:\debug\temp\ntdllhack\ntdll.dll
Debug info file: E:\debug\temp\ntdllhack\wntdll.pdb
Executable:
TimeDateStamp: 55a69e20
Debug info: 2 ( CodeView )
TimeStamp: 55a68c18 Characteristics: 0 MajorVer: 0 MinorVer: 0
Size: 35 RVA: 000e63e0 FileOffset: 000d67e0
CodeView format: RSDS
Signature: {fa9c48f9-c11d-4e08-94b8-970decd92c97} Age: 2
PdbFile: wntdll.pdb
Debug info: 10 ( Unknown )
TimeStamp: 55a68c18 Characteristics: 0 MajorVer: 565 MinorVer: 6526
Size: 4 RVA: 000e63dc FileOffset: 000d67dc
Debug information file:
Format: PDB 7.00
Signature: {b081677d-fc72-4cc4-ac53-992627beea24} Age: 4
Writing to the debug information file...
Result: Success.
0:005> .shell cmd /c copy E:\debug\temp\ntdllhack\wntdll.pdb E:\debug\symbols\wntdll.pdb\FA9C48F9C11D4E0894B8970DECD92C972\wntdll.pdb
1 file(s) copied.
0:005> .reload
Reloading current modules
.............................
0:005> dt nt!_PEB
ntdll!_PEB
+0x000 InheritedAddressSpace : UChar
+0x001 ReadImageFileExecOptions : UChar
...
0:005> !heap -s
LFH Key : 0x219ab08b
Termination on corruption : DISABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-----------------------------------------------------------------------------
Virtual block: 00920000 - 00920000 (size 00000000)
Virtual block: 02c60000 - 02c60000 (size 00000000)
Virtual block: 02e10000 - 02e10000 (size 00000000)
...
Note: using ChkMatch like this has the benefit that you do not need to turn on .symopt- 100, since that option would affect all PDB files, and you would not find potential other symbol issues. If you don't mind using .symopt, you could simply copy an old wntdll.PDB over the new one.
The issue is now fixed according to Microsoft and Microsoft told me that you should clear your symbol cache to get the new PDBs, otherwise Windbg would use the old Symbols which miss the information.
When I do a dump file analysis, I setup symbol path to point to Microsoft symbol server. How does windbg knows that the dump file was generated under what OS and how does it downloads correct symbols for that OS?
To match symbols to binary, WinDbg looks at the thing called Debug Directories. Debug Directories are sections in PE modules (which is a file format used by Windows for all types of executables). Debug Directories simply contain links to types of debug information. If you type command in cmd window link /dump /headers <my_module_name>, it will print out something like this:
...
Debug Directories
Time Type Size RVA Pointer
-------- ------ -------- -------- --------
4CC78FB1 cv 22 00102588 101988 Format: RSDS, {30976E0B-FBF7-45EF-8608-99932F2B791F}, 2, ntdll.pdb
4CC78FB1 ( A) 4 00102584 101984 BB03197E
...
This is output for ntdll.dll. You can see that CV (for CodeView) debug information is contained in ntdll.pdb, and GUID of that PDB has to match the one in the link. That GUID is generated randomly for each module at build time.
Command !lmi in WinDbg will also dump this information, in different format.
What WinDbg does when loading symbols from symbol server is it sends request to symbol server to get file named 'ntdll.pdb' with matching GUID.