Here is the crash i found in the log shown by Windbg. but not able to get anything out of it. please help.
00000000`0008ae08 00000000`76eb219e wow64cpu!CpupSyscallStub+0x2
00000000`0008ae10 00000000`76e7219a wow64cpu!Thunk0ArgReloadState+0x5
00000000`0008aec0 00000000`76e79b14 wow64!RunCpuSimulation+0xa
00000000`0008af10 00000000`76ed840f wow64!Wow64KiUserCallbackDispatcher+0x244
00000000`0008b270 00007ff9`ac2c3044 wow64win!whcbfnINOUTSTYLECHANGE+0xbf
00000000`0008bc50 00000000`76ee9a7a ntdll!KiUserCallbackDispatcherContinue
00000000`0008bcd8 00000000`76ed7a74 wow64win!NtUserSetWindowLongPtr+0xa
00000000`0008bce0 00000000`76e6a44b wow64win!whNtUserSetWindowLong+0x44
00000000`0008bd10 00000000`76eb1dc5 wow64!Wow64SystemServiceEx+0xfb
00000000`0008c5c0 00000000`76e7219a wow64cpu!ServiceNoTurbo+0xb
00000000`0008c670 00000000`76e79b14 wow64!RunCpuSimulation+0xa
00000000`0008c6c0 00000000`76ed880e wow64!Wow64KiUserCallbackDispatcher+0x244
00000000`0008ca20 00007ff9`ac2c3044 wow64win!whcbfnINOUTNCCALCSIZE+0x14e
00000000`0008d470 00000000`76eb2352 ntdll!KiUserCallbackDispatcherContinue
00000000`0008d558 00000000`76eb2318 wow64cpu!CpupSyscallStub+0x2
00000000`0008d560 00000000`76e7219a wow64cpu!Thunk0Arg+0x5
00000000`0008d610 00000000`76e79b14 wow64!RunCpuSimulation+0xa
00000000`0008d660 00000000`76ed5c0a wow64!Wow64KiUserCallbackDispatcher+0x244
00000000`0008d9c0 00007ff9`ac2c3044 wow64win!whcbfnDWORD+0x21a
00000000`0008e3c0 00000000`76ee5a3a ntdll!KiUserCallbackDispatcherContinue
00000000`0008e448 00000000`76ed6df2 wow64win!NtUserMessageCall+0xa
00000000`0008e450 00000000`76ed6b64 wow64win!whNT32NtUserMessageCallCB+0x32
00000000`0008e4a0 00000000`76e6a44b wow64win!whNtUserMessageCall+0xc4
00000000`0008e560 00000000`76eb1dc5 wow64!Wow64SystemServiceEx+0xfb
00000000`0008ee10 00000000`76e7219a wow64cpu!ServiceNoTurbo+0xb
00000000`0008eec0 00000000`76e720d2 wow64!RunCpuSimulation+0xa
00000000`0008ef10 00007ff9`ac2f3a15 wow64!Wow64LdrpInitialize+0x172
00000000`0008f450 00007ff9`ac2d2f1e ntdll!LdrpInitializeProcess+0x1591
00000000`0008f770 00007ff9`ac248ece ntdll!_LdrpInitialize+0x89ffe
00000000`0008f7e0 00000000`00000000 ntdll!LdrInitializeThunk+0xe
As others have mentioned, you have a 64 bit dump of a 32 bit process. We can see this because your excerpt shows 64 bit addresses and it has the wow64 module in the stack.
Typically you should try to get a dump of the correct bitness. The linked SO answer lists several options to get one.
Opening the dump file in the x86 version of WinDbg does not help. Even the x86 version can analyze 64 bit dumps:
The !sw command (load it with .load wow64exts) or the .effmach x86 command can switch the debugger to x86 mode. Note how the command prompt changes:
To show the call stack, you'll now need the wow64exts extension. Also be sure to have set the symbols:
.load wow64exts
.symfix
.reload
Note that there is one call stack per thread, so make sure you get the right one:
~#s
k
In case of a crash, look at the exception:
.exr -1
Related
we have an application that is mostly Go (1.17) that makes a lot of calls through CGo (GCC 7.5) to CUDA on an ARM processor. We occasionally see panics that look like something has done bad things to the heap in the C side. I tried running the whole application under valgrind, but I get too many messages like
==14869== Thread 1:
==14869== Invalid read of size 8
==14869== at 0x4783AC: runtime.startm (proc.go:2508)
==14869== by 0x47890B: runtime.wakep (proc.go:2584)
==14869== by 0x47CF8F: runtime.newproc.func1 (proc.go:4261)
==14869== by 0x4A476B: runtime.systemstack (asm_arm64.s:230)
==14869== by 0x4A465F: runtime.mstart (asm_arm64.s:117)
==14869== Address 0x1fff0001a8 is on thread 1's stack
==14869== 8 bytes below stack pointer
to see anything useful. I am assuming these are false positives, and the Go runtime is not in fact riddled with undefined behaviour. I can't see a flag to suppress that check. Have I missed it? Is there some other way to investigate this problem? I could write test harnesses in C++ but that will change the use pattern which I suspect is key to the problem.
I build the cgo software to an executable with
go build
which produces an executable by the name of the directory you are in. Let's say it's call "mycgoprog"
I then run valgrind against it with this command:
G_SLICE=always-malloc G_DEBUG=gc-friendly valgrind -v --tool=memcheck --leak-check=full --num-callers=40 --log-file=valgrind.log ./mycgoprog
The valgrind.log then contains the all the indiscretions detected by valgrind in the linked c code.
I am making an exe from an existing VB6 project. During Make, VB crashes with the following message
Visual Basic has stopped working
Problem Event Name: APPCRASH
Application Name: vb6.exe
Application Version: 6.0.97.82
Fault Module Name: ntdll.dll
Exception Code: c0000005
I am able to run the project from VB6 without any trouble. The problem occurs when I try to make the exe.
Please could you let me know what could be wrong?
Thanks!
This all needs to be done on the computer with the fault. I cannot load my ntdll.dll as it a different version and the addresses will be different to yours.
Download and install Debugging Tools for Windows
http://msdn.microsoft.com/en-us/windows/hardware/hh852363
Install the Windows SDK but just choose the debugging tools.
Create a folder called Symbols in C:\
This allows WinDbg to get the symbols for your version of ntdll.dll. Start Windbg. File menu - Symbol File Path and enter
srv*C:\symbols*http://msdl.microsoft.com/download/symbols
then
Open ntdll in WinDbg as a crashdump.
It will show the load address.
Type in WinDbg
ln <modloadaddress> + 7c911780
This will give you the nearest symbol to the crash. It probably isn't useful but lets see.
You can also run VB6 under WinDbg (make sure WinDbg is run as admin). When you crash do a stack trace.
Also do an !Analyze when you crash. It is meant for blue screens but will give info on appcrash.
Type in the WinDbg command prompt
!analyze -v
-v stands for Verbose and if the crash was originated by a program, as opposed to hardware or a driver, it will appear in the middle of the listing.
eg
PROCESS_NAME: java.exe
IMAGE_NAME: ntkrnlmp.exe
PROCESS_NAME only appears in the analyze -v output and only if a program originated the call that faulted.
WinDbg Commands
Open as Executable.
windbg -o -g -G c:\windows\system32\cmd.exe /k batfile.bat
You can press F12 to stop it and kb will show the call stack (g continues the program). If there's errors it will also stop and show them.
There is a breakpoint after loading but before any code is run. Press g to continue. Likewise there is a breakpoint after all code has run but before it is unloaded.
Type lm to list loaded modules, x *!* to list the symbols and bp symbolname to set a breakpoint
If programming in VB6 then this environmental variable link=/pdb:none stores the symbols in the dll rather than separate files. Make sure you compile the program with No Optimisations and tick the box for Create Symbolic Debug Info. Both on the Compile tab in the Project's Properties.
Sample output from a nearest symbol search.
Loading Dump File [C:\Windows\System32\ntdll.dll] Symbol search path
is: srvc:\symbolshttp://msdl.microsoft.com/download/symbols
Executable search path is: ModLoad: 4b280000 4b3f9000
C:\Windows\System32\ntdll.dll eax=00000000 ebx=00000000 ecx=00000000
edx=00000000 esi=00000000 edi=00000000 eip=4b280000 esp=00000000
ebp=00000000 iopl=0 nv up di pl nz na pe nc cs=0000 ss=0000
ds=0000 es=0000 fs=0000 gs=0000 efl=00000000
ntdll!__guard_fids_table (ntdll+0x0): 4b280000 4d
dec ebp 0:000> ln 4b280000 + 65534 (4b2e5520)
ntdll!RtlInitializeBitMap+0x14 | (4b2e5540)
ntdll!TpCallbackUnloadDllOnCompletion
Sample stack trace.
You follow what function called what functions. So you read it from the bottom up. It has the first 4 parameters that were passed to the function. You find the debugger starts additional threads so we need to find our program's one.
~
Lists all threads
~<threadid> e <command>
Do a KB on all threads until you find the main one.
0:004> ~0 e kb
ChildEBP RetAddr Args to Child 04bdfc30
75ae325a 04bdfc70 00000000 00000000 USER32!NtUserGetMessage+0xc
04bdfc4c 00895eb6 04bdfc70 00000000 00000000 USER32!GetMessageW+0x2a
04bdfc8c 008a5b41 00890000 00000000 04e2336f notepad!WinMain+0xe6
04bdfd20 74ad3744 7f229000 74ad3720 10fde46e
notepad!WinMainCRTStartup+0x151 04bdfd34 7755a064 7f229000 b0c1107f
00000000 KERNEL32!BaseThreadInitThunk+0x24 04bdfd7c 7755a02f ffffffff
7757d7c9 00000000 ntdll!__RtlUserThreadStart+0x2f 04bdfd8c 00000000
008a59f0 7f229000 00000000 ntdll!_RtlUserThreadStart+0x1b
Assume that 04bdfc70 is an HWnd. Which it is because the documentation says so. But assume it an address of a string. This displays what is there.
ds 775a1300
or to look at the values
db 775a1300
I have a few Windows 7 machines that I am not able to read their memory dumps. I found something that I suspect may be related, but am not positive:
https://twitter.com/aionescu/status/634028737458114560
I also found this: http://support.microsoft.com/kb/2528507
However, the scenario message regarding wow64exts given in the doc is not seen in any of my dumps. I also cannot apply that hotfix at this time to test it. So I'm just looking for some more information or opinions.
I'm able to open any other OS dump as well as my own system's Windows 7 dump, but there are 2 other machines that run Win 7 and it's telling me I have the wrong kernel symbols.
I have tried clearing out my symbol cache, reinstalled the Windows SDK, and also tried to open the dumps on two other machines with the same result. If it matters, the crash is manually created using the scroll lock method.
Symbol path: SRV*c:\symbols*http://msdl.microsoft.com/download/symbols;
Seeing these errors: followed by "Type referenced: nt!_KPRCB"
Does anyone know about the issue mentioned by Alex in the twitter link and if it's possibly related to what I'm seeing?
Update 2015-10-22:
With the Microsoft patch day (2015-10-13) and KB3088195, symbols are available again.
However, symbols for the broken version have not been provided, so below may still be useful.
Microsoft has already published "good" symbols for ntdll in the past, containing type information like _TEB or _KPRCB. Starting from mid of July 2015, Microsoft has still published symbols for ntdll, but not containing that information.
So it depends on the version of ntdll whether you get type information or not. Old dumps referencing an old version of ntdll will download old PDBs containing type information while new dumps reference new versions of ntdll and WinDbg (or any other debugger) downloads PDBs without type information.
Could Microsoft remove type information of "good" symbols retroactively, thus making them "bad"?
Yes. As described in this answer, there is a tool to remove type information from existing PDBs. Doing that and replacing the PDB would result in such an effect.
Can Microsoft publish the "good" version of those PDBs which are currently "bad"?
That's hard to tell, since we don't know whether Microsoft has kept a copy of the "good" version so they could replace the "bad" version on the symbol server with the "good" one. Rebuilding ntdll from the same source code and thus creating new PDBs sounds possible, but the PDB gets a new time stamp and checksum. This can potentially be corrected manually, especially be Microsoft, since they should have the knowledge about the PDB internal format, but IMHO it's unlikely they'll do that. Things may go wrong and MS will hardly have tests to guarantee the correctness of such a thing.
So what can I do?
IMHO you can do nothing to really correct the situation.
You could assume that the types in ntdll have not changed so much. This would allow you to take an older version of wntdll.pdb and the new version of ntdll.dll and apply ChkMatch -m to it. This will copy the timestamp and checksum from the DLL to the PDB. After you did that (in an empty folder), replace the existing wntdll.pdb in your symbols directory with the hacked one.
WinDbg walkthrough (with output shortened to relevant things). I am using the latest version of wntdll.pdb I could find on my PC.
WARNING: doing the following may fix the type information but will likely destroy the correctness of the callstacks. Since any changes in the implementation (which are likely for security fixes) will change the method offsets.
0:005> dt nt!_PEB
*************************************************************************
*** ***
*** Either you specified an unqualified symbol, or your debugger ***
...
*** Type referenced: nt!_PEB ***
*** ***
*************************************************************************
Symbol nt!_PEB not found.
0:005> lm m ntdll
start end module name
773f0000 77570000 ntdll (pdb symbols) e:\debug\symbols\wntdll.pdb\FA9C48F9C11D4E0894B8970DECD92C972\wntdll.pdb
0:005> .shell cmd /c copy C:\Windows\SysWOW64\ntdll.dll e:\debug\temp\ntdllhack\ntdll.dll
1 file(s) copied.
0:005> .shell cmd /c copy "E:\Windows SDk\8.0\Debuggers\x86\sym\wntdll.pdb\B081677DFC724CC4AC53992627BEEA242\wntdll.pdb" e:\debug\temp\ntdllhack\wntdll.pdb
1 file(s) copied.
0:005> .shell cmd /c E:\debug\temp\ntdllhack\chkmatch.exe -m E:\debug\temp\ntdllhack\ntdll.dll E:\debug\temp\ntdllhack\wntdll.pdb
...
Executable: E:\debug\temp\ntdllhack\ntdll.dll
Debug info file: E:\debug\temp\ntdllhack\wntdll.pdb
Executable:
TimeDateStamp: 55a69e20
Debug info: 2 ( CodeView )
TimeStamp: 55a68c18 Characteristics: 0 MajorVer: 0 MinorVer: 0
Size: 35 RVA: 000e63e0 FileOffset: 000d67e0
CodeView format: RSDS
Signature: {fa9c48f9-c11d-4e08-94b8-970decd92c97} Age: 2
PdbFile: wntdll.pdb
Debug info: 10 ( Unknown )
TimeStamp: 55a68c18 Characteristics: 0 MajorVer: 565 MinorVer: 6526
Size: 4 RVA: 000e63dc FileOffset: 000d67dc
Debug information file:
Format: PDB 7.00
Signature: {b081677d-fc72-4cc4-ac53-992627beea24} Age: 4
Writing to the debug information file...
Result: Success.
0:005> .shell cmd /c copy E:\debug\temp\ntdllhack\wntdll.pdb E:\debug\symbols\wntdll.pdb\FA9C48F9C11D4E0894B8970DECD92C972\wntdll.pdb
1 file(s) copied.
0:005> .reload
Reloading current modules
.............................
0:005> dt nt!_PEB
ntdll!_PEB
+0x000 InheritedAddressSpace : UChar
+0x001 ReadImageFileExecOptions : UChar
...
0:005> !heap -s
LFH Key : 0x219ab08b
Termination on corruption : DISABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-----------------------------------------------------------------------------
Virtual block: 00920000 - 00920000 (size 00000000)
Virtual block: 02c60000 - 02c60000 (size 00000000)
Virtual block: 02e10000 - 02e10000 (size 00000000)
...
Note: using ChkMatch like this has the benefit that you do not need to turn on .symopt- 100, since that option would affect all PDB files, and you would not find potential other symbol issues. If you don't mind using .symopt, you could simply copy an old wntdll.PDB over the new one.
The issue is now fixed according to Microsoft and Microsoft told me that you should clear your symbol cache to get the new PDBs, otherwise Windbg would use the old Symbols which miss the information.
I have been scouring the web trying to find an answer to this question, but it seems to be eluding me. I have consulting the following sources before asking this question.
http://www.csn.ul.ie/~caolan/publink/winresdump/winresdump/doc/pefile.html
http://msdn.microsoft.com/en-us/magazine/cc301805.aspx
I understand the PE format (or at least I think I do). Using the command-line debugger (cdb), I would like to be able to disassemble the address where the RVA is to see what the first call is. For a native application (like Notepad), I would expect to see notepad!WinMainCRTStartup, and for a .NET application, I would expect to see a jmp command to the CLR.
Using Notepad as an example, I executed dumpbin /headers on it, and got a value of 3570 for the entry point. When I execute cdb notepad and perform this command - u [base address in memory]+0x3570 - I do not get the WinMainCRTStartup call.
Am I misinterpreting the PE output from dumpbin? How can I know exactly where to look in memory for the starting function of an application?
Edit (1/7/13): I forgot to mention that I am running this on 64-bit Windows 7. If I try to use cdb in Windows XP Mode (to get results from a 32-bit OS), disassembling the AddressOfEntryPoint that I get from an analysis of the PE file gets me the call to WinMainCRTStartup as I would expect. In other words, the exact address I am told to look at contains what I think it should in a 32-bit OS. Does running the application on a 64-bit machine truly make that much of a difference?
Just to add complexity, if I do a !dh on the ImageBaseAddress in the 64-bit OS in cdb, I get the EXACT AddressOfEntryPoint that I need to use.
Use the Microsoft Symbol Server to obtain symbol debugging information. http://support.microsoft.com/kb/311503/en-us
0:001> !dh -a notepad
....
3689 address of entry point
...
00ac0000 image base
...
0:001> u ac3689
notepad!WinMainCRTStartup:
Edit: add dumpbin output (entry point the same offset, image base may be different because ASLR works when image loads in memory):
Microsoft (R) COFF/PE Dumper Version 11.00.50727.1
Copyright (C) Microsoft Corporation. All rights reserved.
Dump of file c:\windows\notepad.exe
PE signature found
File Type: EXECUTABLE IMAGE
FILE HEADER VALUES
14C machine (x86)
4 number of sections
4A5BC60F time date stamp Tue Jul 14 03:41:03 2009
0 file pointer to symbol table
0 number of symbols
E0 size of optional header
102 characteristics
Executable
32 bit word machine
OPTIONAL HEADER VALUES
10B magic # (PE32)
9.00 linker version
A800 size of code
22400 size of initialized data
0 size of uninitialized data
3689 entry point (01003689) _WinMainCRTStartup
Edit 2 add output for x64
dumpbin:
Microsoft (R) COFF/PE Dumper Version 10.00.30319.01
Copyright (C) Microsoft Corporation. All rights reserved.
Dump of file c:\windows\notepad.exe
PE signature found
File Type: EXECUTABLE IMAGE
FILE HEADER VALUES
8664 machine (x64)
6 number of sections
4A5BC9B3 time date stamp Tue Jul 14 03:56:35 2009
0 file pointer to symbol table
0 number of symbols
F0 size of optional header
22 characteristics
Executable
Application can handle large (>2GB) addresses
OPTIONAL HEADER VALUES
20B magic # (PE32+)
9.00 linker version
A800 size of code
25800 size of initialized data
0 size of uninitialized data
3570 entry point (0000000100003570) WinMainCRTStartup
windbg:
0:000> !dh -a notepad
File Type: EXECUTABLE IMAGE
FILE HEADER VALUES
8664 machine (X64)
...
1000 base of code
----- new -----
00000000ff0c0000 image base
...
0:000> u ff0c0000+3570
notepad!WinMainCRTStartup:
I am trying to use GDB to debug a Linux kernel zImage before it is decompressed. The kernel is running on an ARM target and I have a JTAG debugger connected to it with a GDB server stub. The target has to load a boot loader. The boot loader reads the kernel image from flash and puts it in RAM at 0x20008000, then branches to that location.
I have started GDB and connected to the remote target, then I use GDB's add-symbol-file command like so:
add-symbol-file arch/arm/boot/compressed/vmlinux 0x20008000 -readnow
When I set a breakpoint for that address, it does trap at the correct place - right when it branches to the kernel. However, GDB shows the wrong line from the source of arch/arm/boot/compressed/head.S. It's 4 lines behind. How can I fix this?
I also have tried adding the -s section addr option to add-symbol-file with -s .start 0x20008000; this results in exactly the same problem.
There are assembler macros that print out stuff when compiling with low level debug. You have to make sure the macros are appropriate for your board.
linux-latest/arch/arm$ find . -name debug-macro.S | wc
56 56 2306
Find the file for your board and ensure the correct serial port registers are hit. You can instrument the code with out using JTAG. These macros are used in the decompress code. Of course configure with *CONFIG_DEBUG_LL*.
Most likely the ATAGs are not correct or one of the other requirements. Checkout Documentation/arm/Booting to make sure you have registers set properly. Note there is a new requirement with recent kernels to send a dt list.