I have a crash dump to analyze which was taken, allegedly, from a hung system. After running !alanyze -v I get this output from windbg:
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
Unknown bugcheck code (0)
Unknown bugcheck description
Arguments:
Arg1: 0000000000000000
Arg2: 0000000000000000
Arg3: 0000000000000000
Arg4: 0000000000000000
Debugging Details:
------------------
KEY_VALUES_STRING: 1
Key : Analysis.CPU.Sec
Value: 2
Key : Analysis.DebugAnalysisProvider.CPP
Value: Create: 8007007e on MALCOLMMLOCA62C
Key : Analysis.DebugData
Value: CreateObject
Key : Analysis.DebugModel
Value: CreateObject
Key : Analysis.Elapsed.Sec
Value: 2
Key : Analysis.Memory.CommitPeak.Mb
Value: 116
Key : Analysis.System
Value: CreateObject
BUGCHECK_CODE: 0
BUGCHECK_P1: 0
BUGCHECK_P2: 0
BUGCHECK_P3: 0
BUGCHECK_P4: 0
PROCESS_NAME: System
ERROR_CODE: (NTSTATUS) 0x45474150 - <Unable to get error code text>
EXCEPTION_CODE_STR: 45474150
EXCEPTION_PARAMETER1: 4547415045474150
EXCEPTION_PARAMETER2: 4547415045474150
EXCEPTION_PARAMETER3: 4547415045474150
EXCEPTION_PARAMETER4: 4547415045474150
BAD_STACK_POINTER: ffffbc0f13c29900
STACK_TEXT:
ffffbc0f`13c29900 fffff805`1967de62 : ffffd28f`9163f108 ffffbc0f`13c29a40 ffffe600`491a0180 ffffd28f`9163f010 : nt!PpmIdleGuestExecute+0x1d
ffffbc0f`13c29940 fffff805`1967da23 : 00000000`00000000 ffffbc0f`13c29c00 00000000`00000001 00000000`00000001 : nt!PpmIdleExecuteTransition+0x38a
ffffbc0f`13c29b00 fffff805`197a1424 : 00000000`00000000 ffffe600`491a0180 ffffe600`491ab0c0 ffffd28f`950020c0 : nt!PoIdle+0x1b3
ffffbc0f`13c29c60 00000000`00000000 : ffffbc0f`13c2a000 ffffbc0f`13c24000 00000000`00000000 00000000`00000000 : nt!KiIdleLoop+0x54
SYMBOL_NAME: nt!PpmIdleGuestExecute+1d
MODULE_NAME: nt
IMAGE_NAME: ntkrnlmp.exe
STACK_COMMAND: .thread ; .cxr ; kb
BUCKET_ID_FUNC_OFFSET: 1d
FAILURE_BUCKET_ID: 0x0_STACKPTR_ERROR_nt!PpmIdleGuestExecute
OS_VERSION: 10.0.19041.1
BUILDLAB_STR: vb_release
OSPLATFORM_TYPE: x64
OSNAME: Windows 10
FAILURE_ID_HASH: {94784d45-ed21-c95f-fc42-87fec626bbee}
Followup: MachineOwner
On the first examination you can see that error code 45474150 has no correlating entry in NTSTATUS and weirdly all 4 arguments are all just a duplicated over to next 4 bytes value of an exception code, which is suspicious.
Also, if I run .exr -1 I get this:
0: kd> .exr -1
ExceptionAddress: 4547415045474150
ExceptionCode: 45474150
ExceptionFlags: 45474150
NumberParameters: 1162297680
Parameter[0]: 4547415045474150
Parameter[1]: 4547415045474150
Parameter[2]: 4547415045474150
Parameter[3]: 4547415045474150
Parameter[4]: 4547415045474150
Parameter[5]: 4547415045474150
Parameter[6]: 4547415045474150
Parameter[7]: 4547415045474150
Parameter[8]: 4547415045474150
Parameter[9]: 4547415045474150
Parameter[10]: 4547415045474150
Parameter[11]: 4547415045474150
Parameter[12]: 4547415045474150
Parameter[13]: 4547415045474150
Parameter[14]: 4547415045474150
I'm not at all well versed in windows internals and debugging so I'm pretty lost here. Can anyone suggest a debugging pattern for this kind of behavior?
Related
Regularly I need to investigate dumpfiles, always in the same way, and I'd like to automate this. I'm using Windbg as a tool, and I'm thinking of using Windbg scripting.
I have done some first attempts with PYKD, but I don't like the overhead that much, so I've opted for the standard Windbg scripting, but this is getting into a nightmare, let me show you what I want to do:
0:001> kb
# RetAddr : Args to Child : Call Site
00 00007ffc`26272685 : ffffffff`fffffffe 00007ff7`06e563f0 00007ff7`00000000 0000005a`1fb6fd70 : user32!NtUserGetMessage+0xa
01 00007ff7`06d87596 : 00000000`00000008 00007ff7`06e5d048 00000000`00007c1c 0000005a`00000004 : user32!GetMessageW+0x25
02 00007ff7`06d87673 : 0000005a`1f2b3710 00007ff7`06e5c7d0 0000005a`1f2ac270 00000000`00000002 : <Application>!CServiceModule::Run+0x8ee [sourcefile.cpp # 1905]
03 00007ffc`26875ada : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : <Application>!CServiceModule::ServiceMain+0x63 [sourcefile.cpp # 1379]
04 00007ffc`26ef13d2 : 00007ffc`26875aa0 0000005a`1f2ac270 00000000`00000000 00000000`00000000 : sechost!ScSvcctrlThreadA+0x3a
05 00007ffc`270454f4 : 00007ffc`26ef13b0 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x22
06 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x34
0:001> dx Debugger.Sessions[0].Processes[26520].Threads[14812].Stack.Frames[2].SwitchTo();dv /t /v
// The second command "dx ..." happens by clicking on the DML link of the line, corresponding with the line, containing "CServiceModule::Run".
Now, how to do this in a Windbg script?
Launch the kb command and put the result in a variable. (How to do that? I already tried using as and aS but neither seems to work)
If I can't get the whole kb result in one variable, try to use a .foreach, as in following example:
.foreach ( token { kb} )
{
.printf "TEST\n"
}
=> even that is not working: kb has about 8 lines of results, while the word TEST is printed 78 times.
Instead of printing the word "TEST", print the variable token and see what it looks like.
=> How to do that? I already tried .printf "%msu \n" , ${token}, or #$token, ... but nothing is working.
Even if I get this working: how can I do string manipulation, decimal/hexadecimal number conversion, ...?
Not to forget: how do I compile such a script? I'm currently trying to run the script in Windbg, which sometimes gives compilation errors, but those are very unreadable (I add a new line, causing a problem, but the complication error (not a typo) does not even mention that newly added line).
You might say: just have a look at the examples, mentioned under this URL, but I can't find one example of a standard Windbg command, being run, and have its result stored in a variable (which is the first thing I need to do).
In case I get my script working, I might turn this post into a general "Windbg scripting FAQ" for the mentioned (and newly added) questions.
Edit after first answer
I realise that I've mistaken with the kb command: the actual command I need to use is ~* k, giving following result:
(Small remark: I've just a screenshot, instead of a text copy, in order to emphasize the DML hyperlinks)
As you can see, there are some DML results, and I would like to "click" on the line, containing CServiceModule::Run. When I do this by hand, there seems to be a translation towards the following command:
dx Debugger.Sessions[0].Processes[26520].Threads[14812].Stack.Frames[2].SwitchTo()
Here 26520 is the HexToDec conversion of 6798,
and 14812 is the HexToDec conversion of 39DC.
(Both to be retrieved from the screenshot's Id: 6798.39dc)
So I "need" the string manipulation and the HexToDec conversion in order to simulate DML clicking. If, however, you know an easier way to do put this "click" action into a script, I'd very very thankful!
for .foreach space is the delimiter
this command is not meant for reading lines
if you want to print the token use it as is in a alias interpreter
0:001> kb
# ChildEBP RetAddr Args to Child
00 0147fa44 7706f20f 7642d6f7 00000000 00000000 ntdll!DbgBreakPoint
01 0147fa74 75d1ed6c 00000000 0147fac0 770337eb ntdll!DbgUiRemoteBreakin+0x3c
02 0147fa80 770337eb 00000000 7642d643 00000000 kernel32!BaseThreadInitThunk+0xe
03 0147fac0 770337be 7706f1d3 00000000 00000000 ntdll!__RtlUserThreadStart+0x70
04 0147fad8 00000000 7706f1d3 00000000 00000000 ntdll!_RtlUserThreadStart+0x1b
0:001> r $t0 =0 ;.foreach (token { kb } ) { r $t0 = #$t0+1 ; .printf "\"token\" %2d ${token}\n" , #$t0 }
"token" 1 #
"token" 2 ChildEBP
"token" 3 RetAddr
"token" 4 Args
"token" 5 to
"token" 6 Child
"token" 7 00
"token" 8 0147fa44
"token" 9 7706f20f
"token" 10 7642d6f7
"token" 11 00000000
"token" 12 00000000
"token" 13 ntdll!DbgBreakPoint
"token" 14 01
"token" 15 0147fa74
"token" 16 75d1ed6c
"token" 17 00000000
"token" 18 0147fac0
"token" 19 770337eb
"token" 20 ntdll!DbgUiRemoteBreakin+0x3c
"token" 21 02
"token" 22 0147fa80
"token" 23 770337eb
"token" 24 00000000
"token" 25 7642d643
"token" 26 00000000
"token" 27 kernel32!BaseThreadInitThunk+0xe
"token" 28 03
"token" 29 0147fac0
"token" 30 770337be
"token" 31 7706f1d3
"token" 32 00000000
"token" 33 00000000
"token" 34 ntdll!__RtlUserThreadStart+0x70
"token" 35 04
"token" 36 0147fad8
"token" 37 00000000
"token" 38 7706f1d3
"token" 39 00000000
"token" 40 00000000
"token" 41 ntdll!_RtlUserThreadStart+0x1b
i do not understand your further queries so iam assuming your intent below
0:001> $$ putting the result of kb into a variable use javascript
0:001> dx #$foo = Debugger.Utility.Control.ExecuteCommand("kb")
#$foo = Debugger.Utility.Control.ExecuteCommand("kb")
[0x0] : # ChildEBP RetAddr Args to Child
[0x1] : 00 0147fa44 7706f20f 7642d6f7 00000000 00000000 ntdll!DbgBreakPoint
[0x2] : 01 0147fa74 75d1ed6c 00000000 0147fac0 770337eb ntdll!DbgUiRemoteBreakin+0x3c
[0x3] : 02 0147fa80 770337eb 00000000 7642d643 00000000 kernel32!BaseThreadInitThunk+0xe
[0x4] : 03 0147fac0 770337be 7706f1d3 00000000 00000000 ntdll!__RtlUserThreadStart+0x70
[0x5] : 04 0147fad8 00000000 7706f1d3 00000000 00000000 ntdll!_RtlUserThreadStart+0x1b
0:001> dx -r0 #$foo[1]
#$foo[1] : 00 0147fa44 7706f20f 7642d6f7 00000000 00000000 ntdll!DbgBreakPoint
0:001> dx -r0 #$foo[2]
#$foo[2] : 01 0147fa74 75d1ed6c 00000000 0147fac0 770337eb ntdll!DbgUiRemoteBreakin+0x3c
again i don't know what you want to convert to hex or dec
so assumption below
trying to dec a hex from stack as well uppercasing strings
a script like this
function log(a1,a2)
{
host.diagnostics.debugLog(a1 +" " +a2 + "\n");
}
function exec (cmdstr)
{
return host.namespace.Debugger.Utility.Control.ExecuteCommand(cmdstr);
}
function kbtok()
{
var temp = exec("kb")
for ( line of temp )
{
var token = line.split(" ")
for (tok of token)
{
//log(tok)
try {
var num = host.parseInt64(tok,16)
log(tok , num)
}
catch(err) {
log(tok.toUpperCase(), " ")
}
}
}
}
using it like
.load jsprovider
.scriptload ---path----
dx #$scriptcontents.kbtok()
sample output for stack in first paragraph
0:001> dx #$scriptContents.kbtok()
#
CHILDEBP
RETADDR
ARGS
TO
CHILD
00 0
013bfef4 20709108
7706f20f 1996943887
763ed3a7 1983828903
00000000 0
00000000 0
NTDLL!DBGBREAKPOINT
01 1
013bff24 20709156
75d1ed6c 1976692076
00000000 0
013bff70 20709232
770337eb 1996699627
NTDLL!DBGUIREMOTEBREAKIN+0X3C
02 2
013bff30 20709168
770337eb 1996699627
00000000 0
763ed3f3 1983828979
00000000 0
KERNEL32!BASETHREADINITTHUNK+0XE
03 3
013bff70 20709232
770337be 1996699582
7706f1d3 1996943827
00000000 0
00000000 0
NTDLL!__RTLUSERTHREADSTART+0X70
04 4
013bff88 20709256
00000000 0
7706f1d3 1996943827
00000000 0
00000000 0
NTDLL!_RTLUSERTHREADSTART+0X1B
#$scriptContents.kbtok()
you do not compile a script
scripts are interpreted
for a windbg script use $$>a< ---path to script----
for javascript use .scriptload command
I'm trying to boot a Debian Wheezy Image, Ker 3.8 on my BeagleCore (a smaller version of BeagleBone) with TI AM335x Cortex-A8 processor.
I took the Debian Image from beagleboard site.
When I try to boot, on a serial interface for debug, I get this messages:
U-Boot SPL 2016.01-00001-g4eb802e (Jan 13 2016 - 11:14:31)
Trying to boot from MMC
bad magic
U-Boot 2016.01-00001-g4eb802e (Jan 13 2016 - 11:14:31 -0600), Build: jenkins-github_Bootloader-Builder-313
Watchdog enabled
I2C: ready
DRAM: 512 MiB
Reset Source: Power-on reset has occurred.
MMC: OMAP SD/MMC: 0, OMAP SD/MMC: 1
Using default environment
Net: <ethaddr> not set. Validating first E-fuse MAC
Could not get PHY for cpsw: addr 0
cpsw, usb_ether
Press SPACE to abort autoboot in 2 seconds
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
gpio: pin 56 (gpio 56) value is 0
gpio: pin 55 (gpio 55) value is 0
gpio: pin 54 (gpio 54) value is 0
gpio: pin 53 (gpio 53) value is 1
switch to partitions #0, OK
mmc0 is current device
gpio: pin 54 (gpio 54) value is 1
Checking for: /uEnv.txt ...
Checking for: /boot.scr ...
Checking for: /boot/boot.scr ...
Checking for: /boot/uEnv.txt ...
gpio: pin 55 (gpio 55) value is 1
2181 bytes read in 16 ms (132.8 KiB/s)
Loaded environment from /boot/uEnv.txt
Checking if uname_r is set in /boot/uEnv.txt...
gpio: pin 56 (gpio 56) value is 1
Running uname_boot ...
loading /boot/vmlinuz-3.8.13-bone79 ...
5644336 bytes read in 333 ms (16.2 MiB/s)
loading /boot/dtbs/3.8.13-bone79/am335x-boneblack.dtb ...
26118 bytes read in 24 ms (1 MiB/s)
loading /boot/initrd.img-3.8.13-bone79 ...
2905600 bytes read in 179 ms (15.5 MiB/s)
debug: [console=ttyO0,115200n8 capemgr.enable_partno=BB-UART1,BB-UART2,BB-UART4,BB-UART5 capemgr.disable_partno=BB-BONELT-HDMI,BB-BONELT-HDMIN root=UUID=4d8c9d4c-a16d-47ac-a32c-43d0155df072 ro rootfstype=ext4 rootwait coherent_pool=1M quiet init=/lib/systemd/systemd cape_universal=enable] ...
debug: [bootz 0x82000000 0x88080000:2c5600 0x88000000] ...
Kernel image # 0x82000000 [ 0x000000 - 0x562030 ]
## Flattened Device Tree blob at 88000000
Booting using the fdt blob at 0x88000000
Loading Ramdisk to 8fd3a000, end 8ffff600 ... OK
Loading Device Tree to 8fd30000, end 8fd39605 ... OK
Starting kernel ...
Uncompressing Linux... done, booting the kernel.
[ 0.384810] omap2_mbox_probe: platform not supported
[ 0.540541] tps65217-bl tps65217-bl: no platform data provided
[ 0.604330] bone-capemgr bone_capemgr.9: slot #0: No cape found
[ 0.641437] bone-capemgr bone_capemgr.9: slot #1: No cape found
[ 0.678546] bone-capemgr bone_capemgr.9: slot #2: No cape found
[ 0.715656] bone-capemgr bone_capemgr.9: slot #3: No cape found
[ 0.741854] omap_hsmmc mmc.5: of_parse_phandle_with_args of 'reset' failed
[ 0.803809] pinctrl-single 44e10800.pinmux: pin 44e10854 already requested by 44e10800.pinmux; cannot claim for gpio-leds.8
[ 0.815463] pinctrl-single 44e10800.pinmux: pin-21 (gpio-leds.8) status -22
[ 0.822748] pinctrl-single 44e10800.pinmux: could not request pin 21 on device pinctrl-single
[ 0.893233] Unhandled fault: external abort on non-linefetch (0x1008) at 0xe0858c20
[ 0.901225] Internal error: : 1008 [#1] SMP THUMB2
[ 0.906217] Modules linked in:
[ 0.909405] CPU: 0 Not tainted (3.8.13-bone79 #1)
[ 0.914691] PC is at cpts_fifo_read.constprop.1+0x18/0xc4
[ 0.920317] LR is at cpts_systim_read+0x11/0x7c
[ 0.925040] pc : [<c0326468>] lr : [<c0326761>] psr: 000001b3
[ 0.925040] sp : df071db8 ip : 00000000 fp : de231664
[ 0.936993] r10: de231000 r9 : de231758 r8 : c084e0c0
[ 0.942440] r7 : 00000001 r6 : ffffffff r5 : 00000010 r4 : de231670
[ 0.949241] r3 : e0858c00 r2 : 00000001 r1 : de2316d0 r0 : de231670
[ 0.956039] Flags: nzcv IRQs off FIQs on Mode SVC_32 ISA Thumb Segment kernel
[ 0.963925] Control: 50c5387d Table: 80004019 DAC: 00000015
[ 0.969907] Process swapper/0 (pid: 1, stack limit = 0xdf070240)
[ 0.976163] Stack: (0xdf071db8 to 0xdf072000)
[ 0.980699] 1da0: e0858c00 de2316d0
[ 0.989219] 1dc0: de2316bc 35318bf5 00000000 0000001d c052e7a8 c0326761 de2316e8 de2316bc
[ 0.997740] 1de0: 35318bf5 c00611f1 de231670 20000113 de2316e8 c0326927 35318bf5 00000000
[ 1.006259] 1e00: 00000000 00000004 df0d5410 de231000 df0d5400 c0325bab df0d8ac0 de231540
[ 1.014775] 1e20: c0893bb8 0000002b de231540 df0d5400 df0d5410 00000005 00000000 df0d5410
[ 1.023298] 1e40: e0858800 e0858a00 e0858a20 e0858a40 e0858a60 e08588c0 e08588e0 00000008
[ 1.031813] 1e60: 00000001 0000003c 4a102000 4a102000 00002000 00000010 00000001 de231298
[ 1.040338] 1e80: e0858d00 0000000a 00000400 00000002 00000020 00000008 df0d5410 c094362c
[ 1.048868] 1ea0: df0d5410 c08b2c40 00000000 c0829039 00000102 c0846d70 00000000 c02c82b1
[ 1.057381] 1ec0: c02c82a1 c02c7753 00000000 df0d5410 c08b2c40 df0d5444 00000000 c02c78b3
[ 1.065896] 1ee0: c08b2c40 c02c7869 00000000 c02c6887 df049478 df0c6180 c08b2c40 c08a8090
[ 1.074421] 1f00: de23d140 c02c7247 c0753554 c08b2c40 c08b2c40 df070000 c08d4180 00000000
[ 1.082937] 1f20: c0829039 c02c7bb5 00000000 c0833968 df070000 c08d4180 00000000 c0829039
[ 1.091461] 1f40: 00000102 c000867f 00000007 00000007 c088bc98 c0833964 c0833968 00000007
[ 1.099978] 1f60: c0833948 c08d4180 c080d1c9 c0846d70 00000000 c080d6a3 00000007 00000007
[ 1.108503] 1f80: c080d1c9 c0d60fc0 00000000 c04ccfb1 00000000 00000000 00000000 00000000
[ 1.117013] 1fa0: 00000000 c04ccfb7 00000000 c000c8fd 00000000 00000000 00000000 00000000
[ 1.125537] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1.134055] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[ 1.142587] [<c0326468>] (cpts_fifo_read.constprop.1+0x18/0xc4) from [<c0326761>] (cpts_systim_read+0x11/0x7c)
[ 1.153018] [<c0326761>] (cpts_systim_read+0x11/0x7c) from [<c00611f1>] (timecounter_init+0x11/0x1c)
[ 1.162545] [<c00611f1>] (timecounter_init+0x11/0x1c) from [<c0326927>] (cpts_register+0xf3/0x1b8)
[ 1.171894] [<c0326927>] (cpts_register+0xf3/0x1b8) from [<c0325bab>] (cpsw_probe+0x823/0x960)
[ 1.180877] [<c0325bab>] (cpsw_probe+0x823/0x960) from [<c02c82b1>] (platform_drv_probe+0x11/0x14)
[ 1.190222] [<c02c82b1>] (platform_drv_probe+0x11/0x14) from [<c02c7753>] (driver_probe_device+0x53/0x168)
[ 1.200282] [<c02c7753>] (driver_probe_device+0x53/0x168) from [<c02c78b3>] (__driver_attach+0x4b/0x4c)
[ 1.210093] [<c02c78b3>] (__driver_attach+0x4b/0x4c) from [<c02c6887>] (bus_for_each_dev+0x27/0x48)
[ 1.219521] [<c02c6887>] (bus_for_each_dev+0x27/0x48) from [<c02c7247>] (bus_add_driver+0xe3/0x168)
[ 1.228949] [<c02c7247>] (bus_add_driver+0xe3/0x168) from [<c02c7bb5>] (driver_register+0x3d/0xc4)
[ 1.238289] [<c02c7bb5>] (driver_register+0x3d/0xc4) from [<c000867f>] (do_one_initcall+0x1f/0xf4)
[ 1.247630] [<c000867f>] (do_one_initcall+0x1f/0xf4) from [<c080d6a3>] (kernel_init_freeable+0xc3/0x158)
[ 1.257516] [<c080d6a3>] (kernel_init_freeable+0xc3/0x158) from [<c04ccfb7>] (kernel_init+0x7/0x98)
[ 1.266951] [<c04ccfb7>] (kernel_init+0x7/0x98) from [<c000c8fd>] (ret_from_fork+0x11/0x34)
[ 1.275659] Code: 2701 f100 09e8 6823 (6a1a) 07d3
[ 1.280655] ---[ end trace b2036333b4d03ad2 ]---
[ 1.285687] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
With a Debian Jessie image, Ker 4.4, board is booting normally.
Any idea how to solve this, is kindly appreciated.
Thank you.
As per my knowledge there was some issue with CTPS driver in kernel V3.13 (forget the exact kernel version). So it was an open issue. May be with newer kernel version they have fixed it. If you have the source code then try by disabling the CTPS driver CONFIG_TI_CPTS=n.
Sometimes I run my application, it will show [not responding] and be forced to close. I captured a Dump file and open it in WinDbg. Below is the information I get:
Loading unloaded module list
.
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(dd4.1dc): Wake debugger - code 80000007 (first/second chance not available)
eax=00000000 ebx=0018cdb0 ecx=0000000a edx=00000000 esi=00000002 edi=00000000
eip=7708015d esp=0018cd60 ebp=0018cdfc iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
ntdll!NtWaitForMultipleObjects+0x15:
7708015d 83c404 add esp,4
With command ".ecxr"
0.000 > .ecxr
eax=00000000 ebx=0018cdb0 ecx=0000000a edx=00000000 esi=00000002 edi=00000000
eip=7708015d esp=0018cd60 ebp=0018cdfc iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
ntdll!NtWaitForMultipleObjects+0x15:
7708015d 83c404 add esp,4
With command "!analyze -v"
0:000> !analyze -v
***************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
*** ERROR: Symbol file could not be found. Defaulted to export symbols for msvbvm60.dll -
*** ERROR: Module load completed but symbols could not be loaded for AppName.exe
*** ERROR: Symbol file could not be found. Defaulted to export symbols for msjava.dll -
FAULTING_IP:
+0
00000000 ?? ???
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00000000
ExceptionCode: 80000007 (Wake debugger)
ExceptionFlags: 00000000
NumberParameters: 0
BUGCHECK_STR: 80000007
PROCESS_NAME: AppName.exe
OVERLAPPED_MODULE: Address regions for 'msjtes40' and 'msadox.dll' overlap
ERROR_CODE: (NTSTATUS) 0x80000007 - {Kernel Debugger Awakened} the system debugger was awakened by an interrupt.
EXCEPTION_CODE: (HRESULT) 0x80000007 (2147483655) - Operation aborted
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
APP: AppName.exe
DERIVED_WAIT_CHAIN:
Dl Eid Cid WaitType
-- --- ------- --------------------------
0 dd4.1dc Handle
WAIT_CHAIN_COMMAND: ~0s;k;;
BLOCKING_THREAD: 000001dc
DEFAULT_BUCKET_ID: APPLICATION_HANG_HungIn_ExceptionHandler
PRIMARY_PROBLEM_CLASS: APPLICATION_HANG_HungIn_ExceptionHandler
LAST_CONTROL_TRANSFER: from 767615e9 to 7708015d
FAULTING_THREAD: 00000000
STACK_TEXT:
0018cd60 767615e9 00000002 0018cdb0 00000001 ntdll!NtWaitForMultipleObjects+0x15
0018cdfc 76361a2c 0018cdb0 0018ce24 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x100
0018ce44 76364220 00000002 7efde000 00000000 kernel32!WaitForMultipleObjectsExImplementation+0xe0
0018ce60 763880c4 00000002 0018ce94 00000000 kernel32!WaitForMultipleObjects+0x18
0018cecc 76387f83 0018cfa4 00000001 00000001 kernel32!WerpReportFaultInternal+0x186
0018cee0 76387878 0018cfa4 00000001 0018cf7c kernel32!WerpReportFault+0x70
0018cef0 763877f7 0018cfa4 00000001 80471969 kernel32!BasepReportFault+0x20
0018cf7c 7295fa2e 00000000 72a2bd04 0018cfac kernel32!UnhandledExceptionFilter+0x1af
WARNING: Stack unwind information not available. Following frames may be wrong.
0018ff80 00405b66 004068b4 763633aa 7efde000 msvbvm60!Zombie_Release+0x10fd5
0018ff94 77099f72 7efde000 719353d4 00000000 AppName+0x5b66
0018ffd4 77099f45 00405b5c 7efde000 00000000 ntdll!__RtlUserThreadStart+0x70
0018ffec 00000000 00405b5c 7efde000 00000000 ntdll!_RtlUserThreadStart+0x1b
FOLLOWUP_IP:
msvbvm60!Zombie_Release+10fd5
7295fa2e c3 ret
SYMBOL_STACK_INDEX: 8
SYMBOL_NAME: msvbvm60!Zombie_Release+10fd5
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: msvbvm60
IMAGE_NAME: msvbvm60.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 4a5bda6c
STACK_COMMAND: ~0s ; kb
BUCKET_ID: 80000007_msvbvm60!Zombie_Release+10fd5
FAILURE_BUCKET_ID: APPLICATION_HANG_HungIn_ExceptionHandler_80000007_msvbvm60.dll!Zombie_Release
WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/AppName_exe/8_0_0_0/52036491/unknown/0_0_0_0/bbbbbbb4/80000007/00000000.htm?Retriage=1
Followup: MachineOwner
I have a full memory dump but in this instance I don't have a user stack trace database to go with it, I have up to date symbols and the original binaries that go with the dump, normally, I've been able to use the !heap -p -a address to view the call stack at the moment of allocation but this won't work without the user stack trace database.
My question is whether there's another way (albeit less direct approach) to get at the source of this memory leak.
LFH Key : 0x0000005c2dc22701
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-------------------------------------------------------------------------------------
00000000002e0000 00000002 3125248 3122792 3125248 282 378 197 0 7 LFH
0000000000010000 00008000 64 4 64 1 1 1 0 0
0000000000530000 00001002 1088 416 1088 51 10 2 0 0 LFH
0000000000490000 00001002 512 284 512 5 5 1 0 0 LFH
0000000000af0000 00001002 1088 248 1088 2 2 2 0 0 LFH
0000000000c00000 00001002 64 8 64 3 1 1 0 0
0000000000de0000 00001002 512 8 512 3 1 1 0 0
0000000000ac0000 00001002 31616 30356 31616 1810 42 6 0 0 LFH
00000000012c0000 00001002 512 8 512 2 1 1 0 0
0000000002140000 00001003 512 88 512 49 7 1 0 N/A
0000000001ab0000 00001003 512 8 512 5 1 1 0 N/A
00000000022f0000 00001003 512 8 512 5 1 1 0 N/A
0000000002490000 00001003 512 8 512 5 1 1 0 N/A
0000000000d40000 00001003 512 8 512 5 1 1 0 N/A
0000000002690000 00001003 512 8 512 5 1 1 0 N/A
0000000002860000 00001003 512 8 512 5 1 1 0 N/A
0000000002e90000 00001002 512 8 512 2 2 1 0 0
0000000002e10000 00001002 1536 556 1536 40 6 2 0 0 LFH
0000000001b90000 00011002 512 8 512 3 2 1 0 0
00000000033e0000 00001002 512 8 512 3 2 1 0 0
-------------------------------------------------------------------------------------
As you can see from this heap summary (!heap -s), heap 00000000002e0000 has grown pretty large, on closer inspection is can see that 70% of the data is allocated in blocks of size 0x4058, 0x23d1 and 0x10d1 (which is definitely some kind of pattern) so I'm pretty sure I want to investigate that further.
heap # 00000000002e0000
group-by: TOTSIZE max-display: 20
size #blocks total ( %) (percent of total busy bytes)
4058 1ea - 7b2870 (39.56)
23d1 1dc - 42989c (21.39)
10d1 1ed - 20627d (10.40)
c51 1f4 - 180e34 (7.73)
307 25b - 7217d (2.29)
378 1f9 - 6d7b8 (2.20)
188 40e - 63570 (1.99)
c0 59f - 43740 (1.35)
30 12c7 - 38550 (1.13)
28 147e - 333b0 (1.03)
140 22a - 2b480 (0.87)
138 231 - 2abb8 (0.86)
2340 11 - 25740 (0.75)
100 244 - 24400 (0.73)
120 1ea - 22740 (0.69)
78 456 - 20850 (0.65)
1010 12 - 12120 (0.36)
10188 1 - 10188 (0.32)
10008 1 - 10008 (0.32)
4000 4 - 10000 (0.32)
My problem is that I don't know where to go from here, previously I've followed the instructions found here with great success but this time around I don't have a user stack trace database and I can't easily reproduce this pattern but I know the memory dump contains a lot of useful information I'm just not sure how to go about getting at something meaningful from here. Windbg experts? Memory dump analysts? Please advice.
Some blocks, first couple of bytes
0:000> dc 0000000005254b80
00000000`05254b80 52474d45 00000000 050f1c40 00000000 EMGR....#.......
00000000`05254b90 00000000 00000000 00000001 00000001 ................
00000000`05254ba0 00000400 000003ff 0001d4c0 00000001 ................
00000000`05254bb0 524d4954 00000000 051fcd10 00000000 TIMR............
00000000`05254bc0 f7b315d0 000007fe 05254b80 00000000 .........K%.....
00000000`05254bd0 00000000 00000000 05254bd8 00000000 .........K%.....
00000000`05254be0 05254bd8 00000000 05254be8 00000000 .K%......K%.....
00000000`05254bf0 05254be8 00000000 05254bf8 00000000 .K%......K%.....
0:000> dc 00000000051ce640
00000000`051ce640 52474d45 00000000 04f1ab00 00000000 EMGR............
00000000`051ce650 00000000 00000000 00000001 00000001 ................
00000000`051ce660 00000400 000003ff 0001d4c0 00000001 ................
00000000`051ce670 524d4954 00000000 05037070 00000000 TIMR....pp......
00000000`051ce680 f7b315d0 000007fe 051ce640 00000000 ........#.......
00000000`051ce690 00000000 00000000 051ce698 00000000 ................
00000000`051ce6a0 051ce698 00000000 051ce6a8 00000000 ................
00000000`051ce6b0 051ce6a8 00000000 051ce6b8 00000000 ................
0:000> dc 0000000004fdb1f0
00000000`04fdb1f0 52474d45 00000000 04f1b570 00000000 EMGR....p.......
00000000`04fdb200 00000000 00000000 00000001 00000001 ................
00000000`04fdb210 00000400 000003ff 0001d4c0 00000001 ................
00000000`04fdb220 524d4954 00000000 04ed6ba0 00000000 TIMR.....k......
00000000`04fdb230 f7b315d0 000007fe 04fdb1f0 00000000 ................
00000000`04fdb240 00000000 00000000 04fdb248 00000000 ........H.......
00000000`04fdb250 04fdb248 00000000 04fdb258 00000000 H.......X.......
00000000`04fdb260 04fdb258 00000000 04fdb268 00000000 X.......h.......
0:000> dc 0000000001e649b0
00000000`01e649b0 52474d45 00000000 00351270 00000000 EMGR....p.5.....
00000000`01e649c0 00000000 00000000 00000001 00000001 ................
00000000`01e649d0 00000400 000003ff 0001d4c0 00000001 ................
00000000`01e649e0 524d4954 00000000 01e64130 00000000 TIMR....0A......
00000000`01e649f0 f7b315d0 000007fe 01e649b0 00000000 .........I......
00000000`01e64a00 00000000 00000000 01e64a08 00000000 .........J......
00000000`01e64a10 01e64a08 00000000 01e64a18 00000000 .J.......J......
00000000`01e64a20 01e64a18 00000000 01e64a28 00000000 .J......(J......
Use the !heap -flt s on the offending size(s) (with logging to file)
Then manually dump the contents on some of them and try to guess what kind of data they contain.
If you are lucky it’s C++ objects with a vtable address in first DWORD which make them “easy” to recognize.
If not, use dc , dds commands and try to figure out what the contents is.
Another approach is to find types which have corresponding size to those you suspect leaking.
============================Find symbols of spesific size===================================
0:011> dt -v -s a4 <MyDll>!*
Enumerating symbols matching <MyDll>!*, Size = 0xa4
Address Size Symbol
0a4 <MyDll>!NMDATETIMEFORMATW
0a4 <MyDll>!CWinApp
0a4 <MyDll>!CWinApp
==> Check all modules
!for_each_module ".echo ##ModuleName;dt -v -s a4 ${##ModuleName}!*"
You can also try to find heap blocks which has a pointer to a leak suspect
0:008> !heap -srch 09C07058
_HEAP # 02C90000
in HEAP_ENTRY: Size : Prev Flags - UserPtr UserSize - state
0B7DA920: 002c : 002c [01] - 0B7DA928 (00000158) - (busy)
diasymreader!Mod1::`vftable'
We have an outlook addin installed, and while opening some of the html-emails, outlook crashes. If we remove the addin, the emails open properly without outlook getting crashed.
The addin isn't supposed to do anything while opening emails. We can't figure out why outlook is crashing beacause of our addin.
We have also disabled hardware acceleration rendering
We analysed the crashdump and what we got in debuglog is pasted below:
FAULTING_IP:
GdiPlus!ScanOperation::Blend_sRGB_sRGB_MMX+43
4ec5ed94 8907 mov dword ptr [edi],eax
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
.exr 0xffffffffffffffff
ExceptionAddress: 4ec5ed94 (GdiPlus!ScanOperation::Blend_sRGB_sRGB_MMX+0x00000043)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000001
Parameter[1]: a915a784
Attempt to write to address a915a784
DEFAULT_BUCKET_ID: INVALID_POINTER_READ
PROCESS_NAME: OUTLOOK.EXE
ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
EXCEPTION_PARAMETER1: 00000001
EXCEPTION_PARAMETER2: a915a784
WRITE_ADDRESS: a915a784
FOLLOWUP_IP:
GdiPlus!ScanOperation::Blend_sRGB_sRGB_MMX+43
4ec5ed94 8907 mov dword ptr [edi],eax
NTGLOBALFLAG: 0
MANAGED_STACK: !dumpstack -EE
!dumpstack -EE
No export dumpstack found
FAULTING_THREAD: 00001508
PRIMARY_PROBLEM_CLASS: INVALID_POINTER_READ
BUGCHECK_STR: APPLICATION_FAULT_INVALID_POINTER_READ_INVALID_POINTER_WRITE
LAST_CONTROL_TRANSFER: from 4ec5f3a3 to 4ec5ed94
STACK_TEXT:
0013a61c 4ec5f3a3 0000000e 13965ec0 a915a784 GdiPlus!ScanOperation::Blend_sRGB_sRGB_MMX+0x43
0013a638 4ed2aaa2 a915a784 13965ec0 0000000e GdiPlus!EpAlphaBlender::Blend+0x57
0013a690 4ed2ac57 00002000 13965ea8 139663a8 GdiPlus!EpScanGdiDci::DrawScanRecords_Dci+0x1de
0013a6e0 4ecc9c24 00000001 13965ea8 139663a8 GdiPlus!EpScanGdiDci::ProcessBatch_Dci+0x182
0013a704 4ec66d7b 0013a7b8 0013a870 4ec5f2c0 GdiPlus!EpScanGdiDci::EmptyBatch+0xaa
0013a710 4ec5f2c0 0000000e 00000000 4ec619ab GdiPlus!EpScanGdiDci::End+0x1e
0013a71c 4ec619ab 130027f8 13002850 130028d0 GdiPlus!EpScanBufferNative<unsigned long>::~EpScanBufferNative<unsigned long>+0x18
0013a870 4ec9d872 13975eb8 0013a994 130033c8 GdiPlus!DpDriver::DrawImage+0x1ba
0013a8e4 4ec61459 13956260 0013a994 13003548 GdiPlus!DriverMulti::DrawImage+0x78
0013ad40 4ec60d39 0013adb8 13965310 00000003 GdiPlus!GpGraphics::DrvDrawImage+0x2351
0013adf4 4ec609d7 00000000 130021a4 00000000 GdiPlus!GpGraphics::DrawImage+0x215
0013ae60 3d00769f 13956208 13965310 43470000 GdiPlus!GdipDrawImageRectRect+0x1b3
0013aea0 3d0075f3 00000000 00000000 41600000 mshtml!Gdiplus::Graphics::DrawImage+0x62
0013af20 3d123052 0013b0a8 00000000 00e8236d mshtml!XHDC::DrawImage+0x503
0013af80 3d2288bb 0013b0a8 0013b004 0013afe4 mshtml!CImgBitsDIB::StretchBlt+0x2ca
0013b028 3d1607e3 00000640 0013b0a8 1db7d978 mshtml!DrawPlaceHolder+0x33e
0013b0b8 3cf6d4ed 1d99ffd8 0013f658 0013f658 mshtml!CImgHelper::Draw+0x23f
0013b0d0 3cf9b516 0013f658 133f00d0 1d88da2c mshtml!CImgElementLayout::Draw+0x1a
0013b10c 3cf9b461 0013b1a0 0013b164 137215c0 mshtml!CLayout::DrawClient+0x62
0013b4c4 3cf98edf 137e1f88 00000000 00000007 mshtml!CDispLeafNode::DrawSelf+0x432
0013b610 3cf995fe 00000000 137e1f88 00000000 mshtml!CDispNode::Draw+0x217
0013b638 3cf9958e 137e1f88 0013b808 00000000 mshtml!CDispContainer::DrawChildren+0x56
0013b7fc 3cf98edf 137e1f88 1d98a220 00000007 mshtml!CDispContainer::DrawSelf+0x28a
0013b948 3cf995fe 00000000 137e1f88 00000000 mshtml!CDispNode::Draw+0x217
0013b970 3cf9958e 137e1f88 0013bb40 00000000 mshtml!CDispContainer::DrawChildren+0x56
0013bb34 3cf98edf 137e1f88 1d8fb008 00000007 mshtml!CDispContainer::DrawSelf+0x28a
0013bc80 3cf995fe 00000000 137e1f88 00000000 mshtml!CDispNode::Draw+0x217
0013bca8 3cf9958e 137e1f88 0013be78 00000000 mshtml!CDispContainer::DrawChildren+0x56
0013be6c 3cf98edf 137e1f88 1d8fafd0 00000007 mshtml!CDispContainer::DrawSelf+0x28a
0013bfb8 3cf995fe 00000000 137e1f88 00000000 mshtml!CDispNode::Draw+0x217
0013bfe0 3cf9958e 137e1f88 0013c1b0 00000000 mshtml!CDispContainer::DrawChildren+0x56
0013c1a4 3cf98edf 137e1f88 1d8fed68 00000007 mshtml!CDispContainer::DrawSelf+0x28a
0013c2f0 3cf995fe 00000000 137e1f88 00000000 mshtml!CDispNode::Draw+0x217
0013c318 3cf9958e 137e1f88 0013c4e8 00000000 mshtml!CDispContainer::DrawChildren+0x56
0013c4dc 3cf98edf 137e1f88 1db6fc7c 00000007 mshtml!CDispContainer::DrawSelf+0x28a
0013c628 3cf995fe 00000000 137e1f88 00000000 mshtml!CDispNode::Draw+0x217
0013c650 3cf9958e 137e1f88 0013c820 00000000 mshtml!CDispContainer::DrawChildren+0x56
0013c814 3cf98edf 137e1f88 1d889588 00000007 mshtml!CDispContainer::DrawSelf+0x28a
0013c960 3cf995fe 00000000 137e1f88 00000000 mshtml!CDispNode::Draw+0x217
0013c988 3cf9958e 137e1f88 0013cb58 00000000 mshtml!CDispContainer::DrawChildren+0x56
0013cb4c 3cf98edf 137e1f88 137ea79c 00000007 mshtml!CDispContainer::DrawSelf+0x28a
0013cc98 3cf9c490 00000000 137e1f88 00000000 mshtml!CDispNode::Draw+0x217
0013cd48 3ceb0ed7 1d831158 137e1f88 00000000 mshtml!CDispRoot::DrawEntire+0x88
0013f53c 3cf98d12 137215c0 137e1f88 0013f658 mshtml!CDispRoot::DrawRoot+0x319
0013f5ec 3cf9807e 137c3ec8 0013f658 ba0415b1 mshtml!CView::RenderView+0x3b6
0013faa0 3cf7ed29 0000000f 00000000 137c3cd8 mshtml!CDoc::OnPaint+0x5c7
0013fad4 3cfa9457 137c3cd8 0000000f 00000000 mshtml!CServer::OnWindowMessage+0x38f
0013fbfc 3cfa9331 137c3cd8 0000000f 00000000 mshtml!CDoc::OnWindowMessage+0x16c
0013fc28 7e418734 001304b8 0000000f 00000000 mshtml!CServer::WndProc+0x78
0013fc54 7e418816 3cfa92e5 001304b8 0000000f user32!InternalCallWinProc+0x28
0013fcbc 7e42a013 00000000 3cfa92e5 001304b8 user32!UserCallWinProcCheckWow+0x150
0013fcec 7e42a039 ffff0673 001304b8 0000000f user32!CallWindowProcAorW+0x98
0013fd0c 3026f4d5 ffff0673 001304b8 0000000f user32!CallWindowProcW+0x1b
WARNING: Stack unwind information not available. Following frames may be wrong.
0013fd34 7e418734 001304b8 0000000f 00000000 OUTLLIB!FAllowStoreToSend+0x264ff
0013fd60 7e418816 3026f476 001304b8 0000000f user32!InternalCallWinProc+0x28
0013fdc8 7e428ea0 00000000 3026f476 001304b8 user32!UserCallWinProcCheckWow+0x150
0013fe1c 7e428eec 006697a8 0000000f 00000000 user32!DispatchClientMessage+0xa3
0013fe44 7c90e473 0013fe54 00000018 006697a8 user32!__fnDWORD+0x24
0013fe68 7e4194d2 7e428f10 0013fee4 00000000 ntdll!KiUserCallbackDispatcher+0x13
0013feb0 7e418a10 0013fee4 00000000 0013ff04 user32!NtUserDispatchMessage+0xc
0013fec0 30d157fe 0013fee4 0013fee4 300592a7 user32!DispatchMessageW+0xf
0013ff04 3000139e 00000000 00000000 00000000 MSO!Ordinal326+0x21
0013ffc0 7c817077 00380039 00350039 7ffdf000 OUTLOOK+0x139e
0013fff0 00000000 30001084 00000000 00000000 kernel32!BaseProcessStart+0x23
STACK_COMMAND: ~0s; .ecxr ; kb
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: GdiPlus!ScanOperation::Blend_sRGB_sRGB_MMX+43
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: GdiPlus
IMAGE_NAME: GdiPlus.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 4a841b37
FAILURE_BUCKET_ID: INVALID_POINTER_READ_c0000005_GdiPlus.dll!ScanOperation::Blend_sRGB_sRGB_MMX
BUCKET_ID: APPLICATION_FAULT_INVALID_POINTER_READ_INVALID_POINTER_WRITE_GdiPlus!ScanOperation::Blend_sRGB_sRGB_MMX+43
Followup: MachineOwner
I appreciate any help on this issue.