We have a NDIS LWF driver, and on a single machine we get a DPC_WATCHDOG_VIOLATION 133/1 bugcheck when they try to connect to their VPN to connect to the internet. This could be related to our NdisFIndicateReceiveNetBufferLists, as the IRQL is raised to DISPATCH before calling it (and obviously lowered to whatever it was afterward), and that does appear in the output of !dpcwatchdog shown below. This is done due to a workaround for another bug explained here:
IRQL_UNEXPECTED_VALUE BSOD after NdisFIndicateReceiveNetBufferLists?
Now this is the bugcheck:
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
DPC_WATCHDOG_VIOLATION (133)
The DPC watchdog detected a prolonged run time at an IRQL of DISPATCH_LEVEL
or above.
Arguments:
Arg1: 0000000000000001, The system cumulatively spent an extended period of time at
DISPATCH_LEVEL or above. The offending component can usually be
identified with a stack trace.
Arg2: 0000000000001e00, The watchdog period.
Arg3: fffff805422fb320, cast to nt!DPC_WATCHDOG_GLOBAL_TRIAGE_BLOCK, which contains
additional information regarding the cumulative timeout
Arg4: 0000000000000000
STACK_TEXT:
nt!KeBugCheckEx
nt!KeAccumulateTicks+0x1846b2
nt!KiUpdateRunTime+0x5d
nt!KiUpdateTime+0x4a1
nt!KeClockInterruptNotify+0x2e3
nt!HalpTimerClockInterrupt+0xe2
nt!KiCallInterruptServiceRoutine+0xa5
nt!KiInterruptSubDispatchNoLockNoEtw+0xfa
nt!KiInterruptDispatchNoLockNoEtw+0x37
nt!KxWaitForSpinLockAndAcquire+0x2c
nt!KeAcquireSpinLockAtDpcLevel+0x5c
wanarp!WanNdisReceivePackets+0x4bb
ndis!ndisMIndicateNetBufferListsToOpen+0x141
ndis!ndisMTopReceiveNetBufferLists+0x3f0e4
ndis!ndisCallReceiveHandler+0x61
ndis!ndisInvokeNextReceiveHandler+0x1df
ndis!NdisMIndicateReceiveNetBufferLists+0x104
ndiswan!IndicateRecvPacket+0x596
ndiswan!ApplyQoSAndIndicateRecvPacket+0x20b
ndiswan!ProcessPPPFrame+0x16f
ndiswan!ReceivePPP+0xb3
ndiswan!ProtoCoReceiveNetBufferListChain+0x442
ndis!ndisMCoIndicateReceiveNetBufferListsToNetBufferLists+0xf6
ndis!NdisMCoIndicateReceiveNetBufferLists+0x11
raspptp!CallIndicateReceived+0x210
raspptp!CallProcessRxNBLs+0x199
ndis!ndisDispatchIoWorkItem+0x12
nt!IopProcessWorkItem+0x135
nt!ExpWorkerThread+0x105
nt!PspSystemThreadStartup+0x55
nt!KiStartSystemThread+0x28
SYMBOL_NAME: wanarp!WanNdisReceivePackets+4bb
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: wanarp
IMAGE_NAME: wanarp.sys
And this following is the output of !dpcwatchdog, but I still can't find what is causing this bugcheck, and can't find which function is consuming too much time in DISPATCH level which is causing this bugcheck. Although I think this could be related to some spin locking done by wanarp? Could this be a bug with wanarp? Note that we don't use any spinlocking in our driver, and us raising the IRQL should not cause any issue as it is actually very common for indication in Ndis to be done at IRQL DISPATCH.
So How can I find the root cause of this bugcheck? There are no other third party LWF in the ndis stack.
3: kd> !dpcwatchdog
All durations are in seconds (1 System tick = 15.625000 milliseconds)
Circular Kernel Context Logger history: !logdump 0x2
DPC and ISR stats: !intstats /d
--------------------------------------------------
CPU#0
--------------------------------------------------
Current DPC: No Active DPC
Pending DPCs:
----------------------------------------
CPU Type KDPC Function
dpcs: no pending DPCs found
--------------------------------------------------
CPU#1
--------------------------------------------------
Current DPC: No Active DPC
Pending DPCs:
----------------------------------------
CPU Type KDPC Function
1: Normal : 0xfffff80542220e00 0xfffff805418dbf10 nt!PpmCheckPeriodicStart
1: Normal : 0xfffff80542231d40 0xfffff8054192c730 nt!KiBalanceSetManagerDeferredRoutine
1: Normal : 0xffffbd0146590868 0xfffff80541953200 nt!KiEntropyDpcRoutine
DPC Watchdog Captures Analysis for CPU #1.
DPC Watchdog capture size: 641 stacks.
Number of unique stacks: 1.
No common functions detected!
The captured stacks seem to indicate that only a single DPC or generic function is the culprit.
Try to analyse what other processors were doing at the time of the following reference capture:
CPU #1 DPC Watchdog Reference Stack (#0 of 641) - Time: 16 Min 17 Sec 984.38 mSec
# RetAddr Call Site
00 fffff805418d8991 nt!KiUpdateRunTime+0x5D
01 fffff805418d2803 nt!KiUpdateTime+0x4A1
02 fffff805418db1c2 nt!KeClockInterruptNotify+0x2E3
03 fffff80541808a45 nt!HalpTimerClockInterrupt+0xE2
04 fffff805419fab9a nt!KiCallInterruptServiceRoutine+0xA5
05 fffff805419fb107 nt!KiInterruptSubDispatchNoLockNoEtw+0xFA
06 fffff805418a9a9c nt!KiInterruptDispatchNoLockNoEtw+0x37
07 fffff805418da3cc nt!KxWaitForSpinLockAndAcquire+0x2C
08 fffff8054fa614cb nt!KeAcquireSpinLockAtDpcLevel+0x5C
09 fffff80546ba1eb1 wanarp!WanNdisReceivePackets+0x4BB
0a fffff80546be0b84 ndis!ndisMIndicateNetBufferListsToOpen+0x141
0b fffff80546ba7ef1 ndis!ndisMTopReceiveNetBufferLists+0x3F0E4
0c fffff80546bddfef ndis!ndisCallReceiveHandler+0x61
0d fffff80546ba4a94 ndis!ndisInvokeNextReceiveHandler+0x1DF
0e fffff8057c32d17e ndis!NdisMIndicateReceiveNetBufferLists+0x104
0f fffff8057c30d6c7 ndiswan!IndicateRecvPacket+0x596
10 fffff8057c32d56b ndiswan!ApplyQoSAndIndicateRecvPacket+0x20B
11 fffff8057c32d823 ndiswan!ProcessPPPFrame+0x16F
12 fffff8057c308e62 ndiswan!ReceivePPP+0xB3
13 fffff80546c5c006 ndiswan!ProtoCoReceiveNetBufferListChain+0x442
14 fffff80546c5c2d1 ndis!ndisMCoIndicateReceiveNetBufferListsToNetBufferLists+0xF6
15 fffff8057c2b0064 ndis!NdisMCoIndicateReceiveNetBufferLists+0x11
16 fffff8057c2b06a9 raspptp!CallIndicateReceived+0x210
17 fffff80546bd9dc2 raspptp!CallProcessRxNBLs+0x199
18 fffff80541899645 ndis!ndisDispatchIoWorkItem+0x12
19 fffff80541852b65 nt!IopProcessWorkItem+0x135
1a fffff80541871d25 nt!ExpWorkerThread+0x105
1b fffff80541a00778 nt!PspSystemThreadStartup+0x55
1c ---------------- nt!KiStartSystemThread+0x28
--------------------------------------------------
CPU#2
--------------------------------------------------
Current DPC: No Active DPC
Pending DPCs:
----------------------------------------
CPU Type KDPC Function
2: Normal : 0xffffbd01467f0868 0xfffff80541953200 nt!KiEntropyDpcRoutine
DPC Watchdog Captures Analysis for CPU #2.
DPC Watchdog capture size: 641 stacks.
Number of unique stacks: 1.
No common functions detected!
The captured stacks seem to indicate that only a single DPC or generic function is the culprit.
Try to analyse what other processors were doing at the time of the following reference capture:
CPU #2 DPC Watchdog Reference Stack (#0 of 641) - Time: 16 Min 17 Sec 984.38 mSec
# RetAddr Call Site
00 fffff805418d245a nt!KeClockInterruptNotify+0x453
01 fffff80541808a45 nt!HalpTimerClockIpiRoutine+0x1A
02 fffff805419fab9a nt!KiCallInterruptServiceRoutine+0xA5
03 fffff805419fb107 nt!KiInterruptSubDispatchNoLockNoEtw+0xFA
04 fffff805418a9a9c nt!KiInterruptDispatchNoLockNoEtw+0x37
05 fffff805418a9a68 nt!KxWaitForSpinLockAndAcquire+0x2C
06 fffff8054fa611cb nt!KeAcquireSpinLockRaiseToDpc+0x88
07 fffff80546ba1eb1 wanarp!WanNdisReceivePackets+0x1BB
08 fffff80546be0b84 ndis!ndisMIndicateNetBufferListsToOpen+0x141
09 fffff80546ba7ef1 ndis!ndisMTopReceiveNetBufferLists+0x3F0E4
0a fffff80546bddfef ndis!ndisCallReceiveHandler+0x61
0b fffff80546be3a81 ndis!ndisInvokeNextReceiveHandler+0x1DF
0c fffff80546ba804e ndis!ndisFilterIndicateReceiveNetBufferLists+0x3C611
0d fffff8054e384d77 ndis!NdisFIndicateReceiveNetBufferLists+0x6E
0e fffff8054e3811a9 ourdriver+0x4D70
0f fffff80546ba7d40 ourdriver+0x11A0
10 fffff8054182a6b5 ndis!ndisDummyIrpHandler+0x100
11 fffff80541c164c8 nt!IofCallDriver+0x55
12 fffff80541c162c7 nt!IopSynchronousServiceTail+0x1A8
13 fffff80541c15646 nt!IopXxxControlFile+0xC67
14 fffff80541a0aab5 nt!NtDeviceIoControlFile+0x56
15 ---------------- nt!KiSystemServiceCopyEnd+0x25
--------------------------------------------------
CPU#3
--------------------------------------------------
Current DPC: No Active DPC
Pending DPCs:
----------------------------------------
CPU Type KDPC Function
dpcs: no pending DPCs found
Target machine version: Windows 10 Kernel Version 19041 MP (4 procs)
Also note that we also pass the NDIS_RECEIVE_FLAGS_DISPATCH_LEVEL flag to the NdisFIndicateReceiveNetBufferLists, if the current IRQL is dispatch.
Edit1:
This is also the output of !locks and !qlocks and !ready, And the contention count on one of the resources is 49135, is this normal or too high? Could this be related to our issue? The threads that are waiting on it or own it are for normal processes such as chrome, csrss, etc.
3: kd> !kdexts.locks
**** DUMP OF ALL RESOURCE OBJECTS ****
KD: Scanning for held locks.
Resource # nt!ExpTimeRefreshLock (0xfffff80542219440) Exclusively owned
Contention Count = 17
Threads: ffffcf8ce9dee640-01<*>
KD: Scanning for held locks.....
Resource # 0xffffcf8cde7f59f8 Shared 1 owning threads
Contention Count = 62
Threads: ffffcf8ce84ec080-01<*>
KD: Scanning for held locks...............................................................................................
Resource # 0xffffcf8ce08d0890 Exclusively owned
Contention Count = 49135
NumberOfSharedWaiters = 1
NumberOfExclusiveWaiters = 6
Threads: ffffcf8cf18e3080-01<*> ffffcf8ce3faf080-01
Threads Waiting On Exclusive Access:
ffffcf8ceb6ce080 ffffcf8ce1d20080 ffffcf8ce77f1080 ffffcf8ce92f4080
ffffcf8ce1d1f0c0 ffffcf8ced7c6080
KD: Scanning for held locks.
Resource # 0xffffcf8ce08d0990 Shared 1 owning threads
Threads: ffffcf8cf18e3080-01<*>
KD: Scanning for held locks.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Resource # 0xffffcf8ceff46350 Shared 1 owning threads
Threads: ffffcf8ce6de8080-01<*>
KD: Scanning for held locks......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Resource # 0xffffcf8cf0cade50 Exclusively owned
Contention Count = 3
Threads: ffffcf8ce84ec080-01<*>
KD: Scanning for held locks.........................
Resource # 0xffffcf8cf0f76180 Shared 1 owning threads
Threads: ffffcf8ce83dc080-02<*>
KD: Scanning for held locks.......................................................................................................................................................................................................................................................
Resource # 0xffffcf8cf1875cb0 Shared 1 owning threads
Contention Count = 3
Threads: ffffcf8ce89db040-02<*>
KD: Scanning for held locks.
Resource # 0xffffcf8cf18742d0 Shared 1 owning threads
Threads: ffffcf8cee5e1080-02<*>
KD: Scanning for held locks....................................................................................
Resource # 0xffffcf8cdceeece0 Shared 2 owning threads
Contention Count = 4
Threads: ffffcf8ce3a1c080-01<*> ffffcf8ce5625040-01<*>
Resource # 0xffffcf8cdceeed48 Shared 1 owning threads
Threads: ffffcf8ce5625043-02<*> *** Actual Thread ffffcf8ce5625040
KD: Scanning for held locks...
Resource # 0xffffcf8cf1d377d0 Exclusively owned
Threads: ffffcf8cf0ff3080-02<*>
KD: Scanning for held locks....
Resource # 0xffffcf8cf1807050 Exclusively owned
Threads: ffffcf8ce84ec080-01<*>
KD: Scanning for held locks......
245594 total locks, 13 locks currently held
3: kd> !qlocks
Key: O = Owner, 1-n = Wait order, blank = not owned/waiting, C = Corrupt
Processor Number
Lock Name 0 1 2 3
KE - Unused Spare
MM - Unused Spare
MM - Unused Spare
MM - Unused Spare
CC - Vacb
CC - Master
EX - NonPagedPool
IO - Cancel
CC - Unused Spare
IO - Vpb
IO - Database
IO - Completion
NTFS - Struct
AFD - WorkQueue
CC - Bcb
MM - NonPagedPool
3: kd> !ready
KSHARED_READY_QUEUE fffff8053f1ada00: (00) ****------------------------------------------------------------
SharedReadyQueue fffff8053f1ada00: No threads in READY state
Processor 0: No threads in READY state
Processor 1: Ready Threads at priority 15
THREAD ffffcf8ce9dee640 Cid 2054.2100 Teb: 000000fab7bca000 Win32Thread: 0000000000000000 READY on processor 1
Processor 2: No threads in READY state
Processor 3: No threads in READY state
3: kd> dt nt!_ERESOURCE 0xffffcf8ce08d0890
+0x000 SystemResourcesList : _LIST_ENTRY [ 0xffffcf8c`e08d0610 - 0xffffcf8c`e08cf710 ]
+0x010 OwnerTable : 0xffffcf8c`ee6e8210 _OWNER_ENTRY
+0x018 ActiveCount : 0n1
+0x01a Flag : 0xf86
+0x01a ReservedLowFlags : 0x86 ''
+0x01b WaiterPriority : 0xf ''
+0x020 SharedWaiters : 0xffffae09`adcae8e0 Void
+0x028 ExclusiveWaiters : 0xffffae09`a9aabea0 Void
+0x030 OwnerEntry : _OWNER_ENTRY
+0x040 ActiveEntries : 1
+0x044 ContentionCount : 0xbfef
+0x048 NumberOfSharedWaiters : 1
+0x04c NumberOfExclusiveWaiters : 6
+0x050 Reserved2 : (null)
+0x058 Address : (null)
+0x058 CreatorBackTraceIndex : 0
+0x060 SpinLock : 0
3: kd> dx -id 0,0,ffffcf8cdcc92040 -r1 (*((ntkrnlmp!_OWNER_ENTRY *)0xffffcf8ce08d08c0))
(*((ntkrnlmp!_OWNER_ENTRY *)0xffffcf8ce08d08c0)) [Type: _OWNER_ENTRY]
[+0x000] OwnerThread : 0xffffcf8cf18e3080 [Type: unsigned __int64]
[+0x008 ( 0: 0)] IoPriorityBoosted : 0x0 [Type: unsigned long]
[+0x008 ( 1: 1)] OwnerReferenced : 0x0 [Type: unsigned long]
[+0x008 ( 2: 2)] IoQoSPriorityBoosted : 0x1 [Type: unsigned long]
[+0x008 (31: 3)] OwnerCount : 0x1 [Type: unsigned long]
[+0x008] TableSize : 0xc [Type: unsigned long]
3: kd> dx -id 0,0,ffffcf8cdcc92040 -r1 ((ntkrnlmp!_OWNER_ENTRY *)0xffffcf8cee6e8210)
((ntkrnlmp!_OWNER_ENTRY *)0xffffcf8cee6e8210) : 0xffffcf8cee6e8210 [Type: _OWNER_ENTRY *]
[+0x000] OwnerThread : 0x0 [Type: unsigned __int64]
[+0x008 ( 0: 0)] IoPriorityBoosted : 0x1 [Type: unsigned long]
[+0x008 ( 1: 1)] OwnerReferenced : 0x1 [Type: unsigned long]
[+0x008 ( 2: 2)] IoQoSPriorityBoosted : 0x1 [Type: unsigned long]
[+0x008 (31: 3)] OwnerCount : 0x0 [Type: unsigned long]
[+0x008] TableSize : 0x7 [Type: unsigned long]
Thanks for reporting this. I've tracked this down to an OS bug: there's a deadlock in wanarp. This issue appears to affect every version of the OS going back to Windows Vista.
I've filed internal issue task.ms/42393356 to track this: if you have a Microsoft support contract, your rep can get you status updates on that issue.
Meanwhile, you can partially work around this issue by either:
Indicating 1 packet at a time (NumberOfNetBufferLists==1); or
Indicating on a single CPU at a time
The bug in wanarp is exposed when 2 or more CPUs collectively process 3 or more NBLs at the same time. So either workaround would avoid the trigger conditions.
Depending on how much bandwidth you're pushing through this network interface, those options could be rather bad for CPU/battery/throughput. So please try to avoid pessimizing batching unless it's really necessary. (For example, you could make this an option that's off-by-default, unless the customer specifically uses wanarp.)
Note that you cannot fully prevent the issue yourself. Other drivers in the stack, including NDIS itself, have the right to group packets together, which would have the side effect re-batching the packets that you carefully un-batched. However, I believe that you can make a statistically significant dent in the crashes if you just indicate 1 NBL at a time, or indicate multiple NBLs on 1 CPU at a time.
Sorry this is happening to you again! wanarp is... a very old codebase.
Related
I’m using micropython to program an ESP32-CAM and send data via UART to another ESP32, but after the loop runs a couple of times, on average 2, at max 4, I get the error below and the camera robots.
This is my code for reference:
import camera
import time
from machine import UART
from machine import Pin
camera.init()
time.sleep(1)
photo = 10
print('initing UART')
time.sleep(1)
uart1 = UART(1, baudrate=9600, tx=2, rx=16)
time.sleep(1)
print('UART started')
x=range(10)
for count in x:
print('taking picture')
try:
photo = camera.capture()
except Exception as e:
print("An exception occurred while capturing the photo:", e)
time.sleep(1)
print("sending picture")
print(len(str(photo)))
try:
uart1.write(bytes(photo[30]))
except Exception as e:
print("An exception occurred while writing to the UART:", e)
print(count)
time.sleep(15)
This is the error:
Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled.
Core 1 register dump:
PC : 0x400ed39d PS : 0x00060d30 A0 : 0x800e39bb A1 : 0x3ffca5a0
A2 : 0x3f40852c A3 : 0x00000000 A4 : 0x3f951f78 A5 : 0xffffffff
A6 : 0x3f951f40 A7 : 0x3f953753 A8 : 0x800ed330 A9 : 0x3ffca580
A10 : 0x3f950290 A11 : 0x3f407ce0 A12 : 0x3f409cd0 A13 : 0x3f40a220
A14 : 0x3f40a220 A15 : 0x3ffca590 SAR : 0x00000013 EXCCAUSE: 0x0000001c
EXCVADDR: 0xffffffff LBEG : 0x4000c2e0 LEND : 0x4000c2f6 LCOUNT : 0xffffffff
ELF file SHA256: 0000000000000000000000000000000000000000000000000000000000000000
Backtrace: 0x400ed39d:0x3ffca5a0 0x400e39b8:0x3ffca640 0x400df811:0x3ffca670 0x400df83e:0x3ffca690 0x400e0307:0x3ffca6b0 0x400ea98d:0x3ffca740 0x400ea9ad:0x3ffca770 0x400e38fe:0x3ffca790 0x400df811:0x3ffca7c0 0x400ece81:0x3ffca7e0 0x400e39b8:0x3ffca880 0x400df811:0x3ffca8f0 0x400df83e:0x3ffca910 0x40101bc4:0x3ffca930 0x40101e0c:0x3ffca9d0 0x400f4200:0x3ffcaa10 0x4008bf6d:0x3ffcaa50
Rebooting...
ets Jun 8 2016 00:22:57
rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:4948
load:0x40078000,len:9656
load:0x40080400,len:6252
entry 0x400806f4
I (461) psram: This chip is ESP32-D0WD
I (462) spiram: Found 64MBit SPI RAM device
I (462) spiram: SPI RAM mode: flash 40m sram 40m
I (465) spiram: PSRAM initialized, cache is in low/high (2-core) mode.
I (472) cpu_start: Pro cpu up.
I (476) cpu_start: Application information:
I (481) cpu_start: Compile time: Dec 13 2019 08:15:03
I (487) cpu_start: ELF file SHA256: 0000000000000000...
I (493) cpu_start: ESP-IDF: v3.3
I (497) cpu_start: Starting app cpu, entry point is 0x40083830
I (490) cpu_start: App cpu up.
I (1373) spiram: SPI SRAM memory test OK
I (1374) heap_init: Initializing. RAM available for dynamic allocation:
I (1374) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (1380) heap_init: At 3FFBAC10 len 000253F0 (148 KiB): DRAM
I (1386) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM
I (1393) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (1399) heap_init: At 40094998 len 0000B668 (45 KiB): IRAM
I (1406) cpu_start: Pro cpu start user code
I (1411) spiram: Adding pool of 4096K of external SPI memory to heap allocator
I (90) cpu_start: Chip Revision: 1
W (90) cpu_start: Chip revision is higher than the one configured in menuconfig. Suggest to upgrade it.
I (93) cpu_start: Starting scheduler on PRO CPU.
I (0) cpu_start: Starting scheduler on APP CPU.
I (104) spiram: Reserving pool of 32K of internal memory for DMA/internal allocations
I (134) main: Allocated 2815K for micropython heap at 0x3f940020
MicroPython v1.11-665-gfb0141559-kaki5 on 2019-12-13; ESP32 module (camera) with ESP32
By the way, I have seen some other posts regarding the same issue but all of them the programing is happening in c++ or the solution doesn’t work (or I don’t understand)
I apologize if I’m missing something basic or formatting, it’s my first time posting here. Any help appreciated!
I am trying with ESP32-S2 WROOM (full name: LilyGO TTGO T8 ESP32-S2 WROOM) to read the input from an i2s microphone(sph0645), and all I get is either silence or a noise (if the assigned pin is not in the microphone fits) so the connection must be correct, I think. I get 1 error message namely E (10715) AUDIO_ELEMENT: [wav] Element already stopped. This error message is nowhere to be found on the espressif forum so I have become a bit hopeless: P also good to know that this is my first post on this topic. So if you know a little more about how to make these two components (i2s microphone and esp32-s2) work, feel free!
To complete the story, this is the complete log.
(64) boot: ESP-IDF v4.2-dirty 2nd stage bootloader
I (64) boot: compile time 15:26:45
I (64) boot: chip revision: 0
I (66) boot.esp32s2: SPI Speed : 80MHz
I (71) boot.esp32s2: SPI Mode : DIO
I (76) boot.esp32s2: SPI Flash Size : 4MB
I (81) boot: Enabling RNG early entropy source...
I (86) boot: Partition Table:
I (90) boot: ## Label Usage Type ST Offset Length
I (97) boot: 0 nvs WiFi data 01 02 00009000 00006000
I (104) boot: 1 phy_init RF data 01 01 0000f000 00001000
I (112) boot: 2 factory factory app 00 00 00010000 00100000
I (119) boot: End of partition table
I (124) esp_image: segment 0: paddr=0x00010020 vaddr=0x3f000020 size=0x0dafc ( 56060) map
I (144) esp_image: segment 1: paddr=0x0001db24 vaddr=0x3ffbf260 size=0x01f70 ( 8048) load
I (147) esp_image: segment 2: paddr=0x0001fa9c vaddr=0x40024000 size=0x00404 ( 1028) load
0x40024000: _WindowOverflow4 at C:/Users/kraan/esp/esp-idf/components/freertos/xtensa/xtensa_vectors.S:1730I (153) esp_image: segment 3: paddr=0x0001fea8 vaddr=0x40024404 size=0x00170 ( 368) load
I (162) esp_image: segment 4: paddr=0x00020020 vaddr=0x40080020 size=0x2a02c (172076) map
0x40080020: _stext at ??:?I (206) esp_image: segment 5: paddr=0x0004a054 vaddr=0x40024574 size=0x0ace4 ( 44260) load
I (225) boot: Loaded app from partition at offset 0x10000
I (225) boot: Disabling RNG early entropy source...
I (225) cache: Instruction cache : size 8KB, 4Ways, cache line size 32Byte
I (233) cpu_start: Pro cpu up.
I (236) cpu_start: Application information:
I (241) cpu_start: Project name: record_wav
I (246) cpu_start: App version: 1
I (251) cpu_start: Compile time: Mar 4 2021 15:26:18
I (257) cpu_start: ELF file SHA256: f689f9b5ef13ab60...
I (263) cpu_start: ESP-IDF: v4.2-dirty
I (268) cpu_start: Single core mode
D (272) memory_layout: Checking 3 reserved memory ranges:
D (278) memory_layout: Reserved memory range 0x3ffb4000 - 0x3ffbf258
D (284) memory_layout: Reserved memory range 0x3ffbf260 - 0x3ffc1a20
D (291) memory_layout: Reserved memory range 0x3ffffa10 - 0x40000000
D (297) memory_layout: Building list of available memory regions:
D (303) memory_layout: Available memory region 0x3ffc1a20 - 0x3ffc4000
D (310) memory_layout: Available memory region 0x3ffc4000 - 0x3ffc8000
D (316) memory_layout: Available memory region 0x3ffc8000 - 0x3ffcc000
D (323) memory_layout: Available memory region 0x3ffcc000 - 0x3ffd0000
D (330) memory_layout: Available memory region 0x3ffd0000 - 0x3ffd4000
D (336) memory_layout: Available memory region 0x3ffd4000 - 0x3ffd8000
D (343) memory_layout: Available memory region 0x3ffd8000 - 0x3ffdc000
D (349) memory_layout: Available memory region 0x3ffdc000 - 0x3ffe0000
D (356) memory_layout: Available memory region 0x3ffe0000 - 0x3ffe4000
D (363) memory_layout: Available memory region 0x3ffe4000 - 0x3ffe8000
D (369) memory_layout: Available memory region 0x3ffe8000 - 0x3ffec000
D (376) memory_layout: Available memory region 0x3ffec000 - 0x3fff0000
D (382) memory_layout: Available memory region 0x3fff0000 - 0x3fff4000
D (389) memory_layout: Available memory region 0x3fff4000 - 0x3fff8000
D (396) memory_layout: Available memory region 0x3fff8000 - 0x3fffc000
D (402) memory_layout: Available memory region 0x3fffc000 - 0x3ffffa10
I (409) heap_init: Initializing. RAM available for dynamic allocation:
D (416) heap_init: New heap initialised at 0x3ffc1a20
I (421) heap_init: At 3FFC1A20 len 0003A5E0 (233 KiB): DRAM
I (427) heap_init: At 3FFFC000 len 00003A10 (14 KiB): DRAM
I (433) cpu_start: Pro cpu start user code
D (491) clk: RTC_SLOW_CLK calibration value: 5765894
D (497) intr_alloc: Connected src 49 to int 2 (cpu 0)
D (497) intr_alloc: Connected src 73 to int 10 (cpu 0)
D (497) intr_alloc: Connected src 28 to int 3 (cpu 0)
D (503) FLASH_HAL: extra_dummy: 0
D (506) spi_flash: trying chip: issi
D (510) spi_flash: trying chip: gd
D (513) spi_flash: trying chip: mxic
D (517) spi_flash: trying chip: generic
I (521) spi_flash: detected chip: generic
I (526) spi_flash: flash io: dio
I (530) cpu_start: Starting scheduler on PRO CPU.
D (535) heap_init: New heap initialised at 0x3fffc000
D (535) intr_alloc: Connected src 17 to int 9 (cpu 0)
I (545) REC_WAV_SDCARD: [ 1 ] Mount sdcard
D (545) intr_alloc: Connected src 23 to int 12 (cpu 0)
D (555) intr_alloc: Connected src 34 to int 13 (cpu 0)
I (605) REC_WAV_SDCARD: [ 2 ] Start codec chip
I (605) REC_WAV_SDCARD: [3.0] Create audio pipeline for recording
I (605) REC_WAV_SDCARD: [3.1] Create fatfs stream to write data to sdcard
I (615) REC_WAV_SDCARD: [3.2] Create i2s stream to read audio data from codec chip
D (625) intr_alloc: Connected src 35 to int 19 (cpu 0)
I (635) REC_WAV_SDCARD: [3.3] Create wav encoder to encode wav format
I (635) REC_WAV_SDCARD: [3.4] Register all elements to audio pipeline
I (645) REC_WAV_SDCARD: [3.5] Link it together [codec_chip]-->i2s_stream-->wav_encoder-->fatfs_stream-->[sdcard]
I (655) REC_WAV_SDCARD: [3.6] Set up uri (file as fatfs_stream, wav as wav encoder)
I (665) REC_WAV_SDCARD: [ 4 ] Set up event listener
I (665) REC_WAV_SDCARD: [4.1] Listening event from pipeline
I (675) REC_WAV_SDCARD: [4.2] Listening event from peripherals
I (685) REC_WAV_SDCARD: [ 5 ] Start audio_pipeline
I (695) REC_WAV_SDCARD: [ 6 ] Listen for all pipeline events, record for 10 Seconds
I (1715) REC_WAV_SDCARD: [ * ] Recording ... 1
I (2715) REC_WAV_SDCARD: [ * ] Recording ... 2
I (3715) REC_WAV_SDCARD: [ * ] Recording ... 3
I (4715) REC_WAV_SDCARD: [ * ] Recording ... 4
I (5715) REC_WAV_SDCARD: [ * ] Recording ... 5
I (6715) REC_WAV_SDCARD: [ * ] Recording ... 6
I (7715) REC_WAV_SDCARD: [ * ] Recording ... 7
I (8715) REC_WAV_SDCARD: [ * ] Recording ... 8
I (9715) REC_WAV_SDCARD: [ * ] Recording ... 9
I (10715) REC_WAV_SDCARD: [ * ] Recording ... 10
I (10715) REC_WAV_SDCARD: [ 7 ] Stop audio_pipeline
W (10715) AUDIO_ELEMENT: IN-[wav] AEL_IO_ABORT
E (10715) AUDIO_ELEMENT: [wav] Element already stopped
W (10725) AUDIO_ELEMENT: IN-[file] AEL_IO_ABORT
W (10735) AUDIO_PIPELINE: There are no listener registered
W (10735) AUDIO_ELEMENT: [file] Element has not create when AUDIO_ELEMENT_TERMINATE
W (10745) AUDIO_ELEMENT: [i2s] Element has not create when AUDIO_ELEMENT_TERMINATE
W (10755) AUDIO_ELEMENT: [wav] Element has not create when AUDIO_ELEMENT_TERMINATE
Can't understand how Windows Memory Manager works.
I look at the attached user process (dbgview.exe).
It is WOW64-process. At the specified address (0x76560000) there is .text section of the kernel32.dll module (also WOW64).
Why there is no PTE and other tables in the process page table pointing to those virtual address?
kd> db 76560000
00000000`76560000 8b ff 55 8b ec 51 56 57-33 f6 89 55 fc 56 68 80 ..U..QVW3..U.Vh.
<...>
kd> !pte 76560000
VA 0000000076560000
PXE at FFFFF6FB7DBED000 PPE at FFFFF6FB7DA00008 PDE at FFFFF6FB40001D90 PTE at FFFFF680003B2B00
Unable to get PXE FFFFF6FB7DBED000
kd> db FFFFF680003B2B00
fffff680`003b2b00 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ???????????????
<...>
I know that pages will be allocated after first access (with page fault) have occured, but why there is no protype PTE too?
Firstly, translate an arbitrary virtual address to physical using !vtop to see the dirbase of the process in the process of translation, or use !process to find the dirbase of the process:
lkd> .process /p fffffa8046a2e5f0
Implicit process is now fffffa80`46a2e5f0
lkd> .context 77fa90000
lkd> !vtop 0 13fe60000
Amd64VtoP: Virt 00000001`3fe60000, pagedir 7`7fa90000
Amd64VtoP: PML4E 7`7fa90000
Amd64VtoP: PDPE 1`c2e83020
Amd64VtoP: PDE 7`84e04ff8
Amd64VtoP: PTE 4`be585300
Amd64VtoP: Mapped phys 6`3efae000
Virtual address 13fe60000 translates to physical address 63efae000.
Then find that physical frame in the PFN database (in this case the physical page for PML4 (cr3 page aka. dirbase) is 77fa90 with full physical address 77fa90000:
lkd> !pfn 77fa90
PFN 0077FA90 at address FFFFFA80167EFB00
flink FFFFFA8046A2E5F0 blink / share count 00000005 pteaddress FFFFF6FB7DBEDF68
reference count 0001 used entry count 0000 Cached color 0 Priority 0
restore pte 00000080 containing page 77FA90 Active M
Modified
The address FFFFF6FB7DBED000 is therefore the virtual address of the PML4 page and FFFFF6FB7DBEDF68 is the virtual address of the PML4E self reference entry (1ed*8 = f68).
FFFFF6FB7DBED000 = 1111111111111111111101101111101101111101101111101101000000000000
1111111111111111 111101101 111101101 111101101 111101101 000000000000
The PML4 can only be at a virtual address where the PML4E, PDTPE, PDE and PTE index are the same, so there are actually 2^9 different combinations of that and windows 7 always selects 0x1ed i.e. 111101101. The reason for this is because the PML4 contains a PML4 that points to itself i.e. the physical frame of the PML4, so it will need to keep indexing to that same location at every level of the hierarchy.
The PML4, being a page table page, must reside in the kernel, and kernel addresses are high-canonical, i.e. prefixed with 1111111111111111, and kernel addresses begin with 00001 through 11111 i.e. from 08 to ff
The range of possible addresses that a 64 bit OS that uses 8TiB for user address space can place it at is therefore 31*(2^4) = 496 different possible locations and not actually 2^9:
1111111111111111 000010000 000010000 000010000 000010000 000000000000
1111111111111111 111111111 111111111 111111111 111111111 000000000000
I.e. the first is FFFF080402010000, the second is FFFF088442211000, the last is FFFFFFFFFFFFF000.
Note:
Up until Windows 10 TH2, the magic index for the Self-Reference PML4 entry was 0x1ed as mentioned above. But what about Windows 10 from 1607? Well Microsoft uped their game, as a constant battle for improving Windows security the index is randomized at boot-time, so 0x1ed is now one of the 512 [sic. (496)] possible values (i.e. 9-bit index) that the Self-Reference entry index can have. And side effect, it also broke some of their own tools, like the !pte2va WinDbg command.
0xFFFFF68000000000 is the address of the first PTE in the first page table page, so basically MmPteBase, except because on Windows 10 1607 the PML4E can be an other than 0x1ed, the base is not always 0xFFFFF68000000000 as a result, and it uses a variable nt!MmPteBase to know instantly where the base of the page table page allocations begins. Previously, this symbol does not exist in ntoskrnl.exe, because it has a hardcoded base 0xFFFFF68000000000. The address of the first and last page table page is going to be:
first last
* pml4e_offset : 0x1ed 0x1ed
* pdpe_offset : 0x000 0x1ff
* pde_offset : 0x000 0x1ff
* pte_offset : 0x000 0x1ff
* offset : 0x000 0x000
This gives 0xFFFFF68000000000 for the first and 0xFFFFF6FFFFFFF000 for the last page table page when the PML4E index is 0x1ed. PDEs + PDPTEs + PML4Es + PTEs are assigned in this range.
Therefore, to be able to translate a virtual address to its PTE virtual address (and !pte2va is the reverse of this), you affix 111101101 to the start of the virtual address and then you truncate the last 12 bits (the page offset, which is no longer useful) and then you times it by 8 bytes (the PTE size) (i.e. add 3 zeroes to the end, which creates a new page offset from the last level index into the page that contains the PTEs times the size of a PTE structure). Concatenating the PML4E index to the start simply causes it to loop back one time such that you actually get the PTE rather than what the PTE points to. Concatenating it to the start is the same thing as adding it to MmPteBase.
Here is simple C++ code to do it:
// pte.cpp
#include<iostream>
#include<string>
int main(int argc, char *argv[]) {
unsigned long long int input = std::stoull(argv[1], nullptr, 16);
long long int ptebase = 0xFFFFF68000000000;
long long int pteaddress = ptebase + ((input >> 12) << 3);
std::cout << "0x" << std::hex << pteaddress;
}
C:\> pte 13fe60000
0xfffff680009ff300
To get the PDE virtual address you have to affix it twice and then truncate the last 21 bits and then times by 8. This is how !pte is supposed to work, and is the opposite of !pte2va.
Similarly, PDEs + PDPTEs + PML4Es are assigned in the range:
first last
* pml4e_offset : 0x1ed 0x1ed
* pdpe_offset : 0x1ed 0x1ed
* pde_offset : 0x000 0x1ff
* pte_offset : 0x000 0x1ff
* offset : 0x000 0x000
Because when you get to 0x1ed for the pdpte offset within the page table page range, all of a sudden, you are looping back in the PML4 once again, so you get the PDE.
If it says there is no PTE for an address within a virtual page for which the corresponding physical frame is shown to be part of the working set by VMMap, then you might be experiencing my issue, where you need to use .process /P if you're doing live kernel debugging (local or remote) to explicitly tell the debugger that you want to translate user and kernel addresses in the context of the process and not the debugger.
I have found that since Windows 10 Anniversary Update (1607, 10.0.14393) PML4 table had been randomized to mitigate kernel heap spraying.
It means that probably Page Table is not placed at 0xFFFFF6800000.
I'm trying to inspect a kernel module that utilizes usb, and so from the module itself I'm writing a message to ftrace using trace_printk; and then I wanted to inspect when does a USB Bulk Out URB Submit appear in the system after that.
The problem is that on my Ubuntu Lucid 11.04 (kernel 2.6.38-16), there are only local and global clocks in ftrace - and although their resolution is the same (microseconds) as the timestamps by usbmon, their values differ significantly.
So not knowing any better (as I couldn't find anywhere else talking about this), what I did was attempt to redirect usbmon to trace_marker, using cat:
# ... activate ftrace here ...
usbpid=$(sudo bash -c 'cat /sys/kernel/debug/usb/usbmon/2u > /sys/kernel/debug/tracing/trace_marker & echo $!')
sleep 3 # do test, etc.
sudo kill $usbpid
# ... deactivate ftrace here...
... and then, when I read from /sys/kernel/debug/tracing/trace, I get a log with problematic timestamps (see below). So what I'd like to know is:
Is there a way to make usbmon have it's messages appear directly in /debug/tracing/trace, instead of in /debug/usb/usbmon/2u ? (not that I can see, but I'd like to have this confirmed)
If not, is there a better way to "directly" redirect output of /sys/kernel/debug/usb/usbmon/2u without any possible overhead/buffering issues of cat and/or shell redirection?
If not, is there some sort of an algorithm, where I could use the extra usbmon timestamp, to somehow "correct" the position of these events in the kernel timestamp domain? (see example below)
Here is a brief example snippet of a /sys/kernel/debug/tracing/trace log I got:
<idle>-0 [000] 44989.403572: my_kernel_function: 1 00 00 64 1 64 5
<...>-29765 [000] 44989.403918: my_kernel_function: 1 00 00 64 2 128 2
<...>-29787 [000] 44989.404202: 0: f1f47280 3237249791 S Bo:2:002:2 -115 64 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<...>-29787 [000] 44989.404234: 0: f1f47080 3237250139 S Bo:2:002:2 -115 64 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<idle>-0 [000] 44989.404358: my_kernel_function: 1 00 00 64 3 192 4
<...>-29787 [000] 44989.404402: 0: f1f47c00 3237250515 S Bo:2:002:2 -115 64 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
So when the kernel timestamp is 44989.404202, the usbmon timestamp is 3237.249791 (= 3237249791/1e6); neither the seconds nor the microseconds part match. To make it a bit easier on the eyes, here's the same snippet with only time information remaining:
(1) 44989.403572 MYF 0
(2) 44989.403918 MYF 0.000346
(3) 44989.404202 USB | 0 3237.249791 0
(4) 44989.404234 USB | 0.000032 3237.250139 0.000348
(5) 44989.404358 MYF 0.000440 | |
(6) 44989.404402 USB 0.000168 3237.250515 0.000376
So judging by the kernel timestamps, 32 μs expired between event (3) and event (4) - but judging by the usbmon timestamps, 348 μs expired between the same events! Whom to trust now?!
Now, if we assume that the usbmon timestamps are more correct for those messages, given they got "printed" before they ended up in the ftrace buffer to begin with - we could assume that the first usb message (3) may have been scheduled right after (1) executed, but something preempted it - and so the second USB message (4) triggered the "printout" (or rather, "entry") of both (3) and (4) in the ftrace buffer (which is why their kernel timestamps are so clcse together?)
So, if I assume (4) is the more correct one, I can try push (3) back for 348 μs:
(1) 44989.403572 MYF 0
(3) 44989.403854 USB | 0 3237.249791 0
(2) 44989.403918 MYF 0.000346 | |
(4) 44989.404234 USB | 0.000380 3237.250139 0.000348
(5) 44989.404358 MYF 0.000440 | |
(6) 44989.404402 USB 0.000168 3237.250515 0.000376
... and that sort of looks better (also USB now fires 282 μs, 316 μs, and 44 μs after MYF) - for first and second MYF/USB pairs (if that is indeed how they behave); but then the third step doesn't really match, and so on... Cannot really think of an algorithm to help me adjust the USB events position according to the data in the usbmon timestamp...
While the best approach for redirecting usbmon output to ftrace is still an open question, I got an answer about correlating their timestamps from this thread:
Using both usbmon and ftrace? [linux-usb mailing list]
You can call the following
subroutine to get a usbmon-style timestamp value, which can then be
added to an ftrace message or simply printed in the kernel log:
#include <linux/time.h>
static unsigned usbmon_timestamp(void)
{
struct timeval tval;
unsigned stamp;
do_gettimeofday(&tval);
stamp = tval.tv_sec & 0xFFF;
stamp = stamp * 1000000 + tval.tv_usec;
return stamp;
}
For example,
pr_info("The usbmon time is: %u\n", usbmon_timestamp());
I have a windows service running on Windows Server 2003. The service crashes ocassionlly. I tried analysis error through DebugDiag and got following information. Can anyone help me to understand the problem. The service uses GSM libaray for SMS service.
Error:
In EnfieldSMSService__PID__1764__Date__01_21_2011__Time_09_34_31AM__554__Second_Chance_Exception_C0000005.
dmp the assembly instruction at 0x00bb29d1 which does not correspond to any
known native module in the process has caused an access violation exception
(0xC0000005) when trying to read from memory location 0x00000008 on thread 10
Report for
>EnfieldSMSService__PID__1764__Date__01_21_2011__Time_09_34_31AM__554__Second_Chance_Exception_C0000005.
dmp
Type of Analysis Performed Crash Analysis
Machine Name REALTRACSERVER
Operating System Windows Server 2003 Service Pack 2
Number Of Processors 4
Process ID 1764
Process Image C:\Program Files\Default Company Name\EnfieldSMSSetUp\
EnfieldSMSService.exe
System Up-Time 7 day(s) 18:30:55
Process Up-Time 00:05:24
Thread 10 - System ID 3632
Entry point mscorwks!CreateApplicationContext+bbef
Create time 1/21/2011 9:34:31 AM
Time spent in user mode 0 Days 0:0:0.0
Time spent in kernel mode 0 Days 0:0:0.0
Function Arg 1 Arg 2 Arg 3 Source
0x00bb29d1 01041e54 00b8f8c4 792d6cf6
0x00bb11c7 01041ee0 00b8f8d8 792e019f
mscorlib_ni+216cf6 00b8f91c 01041ee0 01041eac
mscorlib_ni+22019f 01041eac 00000000 001df020
mscorlib_ni+216c74 00000217 7c82b02a 00b8f980
mscorwks+1b4c 00b8f9d0 00000000 00b8f9a0
mscorwks!DllUnregisterServerInternal+6195 00b8f9d0 00000000
00b8f9a0
mscorwks!CoUninitializeEE+2e95 7924290c 00b8fc14 00b8fb4c
mscorwks!CoUninitializeEE+2ec8 7924290c 00b8fc14 00b8fb4c
mscorwks!CoUninitializeEE+2ee6 00b8fb4c 95f5cb6e 001df020
mscorwks!CorExitProcess+1e4d 00b8fe50 00000000 00000000
mscorwks!CoUninitializeEE+4df3 00b8fdc4 00b8fd70 79f7759b
mscorwks!CoUninitializeEE+4d8f 00b8fdc4 95f5ca02 00000000
mscorwks!CoUninitializeEE+4cb5 00b8fdc4 00000001 00000000
mscorwks!CoUninitializeEE+4e41 00000001 79f3d6e9 00b8fe50
mscorwks!CorExitProcess+1c1e 00000001 79f3d6e9 00b8fe50
mscorwks!CorExitProcess+1cf8 001e2f68 00000001 00000001
mscorwks!CreateApplicationContext+bc35 0019cf88 00000000 00000000
kernel32!GetModuleHandleA+df 79f91fcf 0019cf88 00000000
Detailed stack corruption analysis for thread 10
Call stack with StackWalk
Index Return Address
1 0x00bb29d1
2 0x00bb11c7
3 mscorlib_ni+216cf6
4 mscorlib_ni+22019f
5 mscorlib_ni+216c74
6 mscorwks+1b4c
7 mscorwks!DllUnregisterServerInternal+6195
8 mscorwks!CoUninitializeEE+2e95
9 mscorwks!CoUninitializeEE+2ec8
10 mscorwks!CoUninitializeEE+2ee6
11 mscorwks!CorExitProcess+1e4d
12 mscorwks!CoUninitializeEE+4df3
13 mscorwks!CoUninitializeEE+4d8f
14 mscorwks!CoUninitializeEE+4cb5
15 mscorwks!CoUninitializeEE+4e41
16 mscorwks!CorExitProcess+1c1e
17 mscorwks!CorExitProcess+1cf8
18 mscorwks!CreateApplicationContext+bc35
19 kernel32!GetModuleHandleA+df
Call stack - Heuristic
Index Stack Address Child EBP Return Address Destination
1 0x00000000 0x00b8f8ac 0x00bb29d1 0x00000000
2 0x00b8f8b0 0x00b8f8b8 0x00bb11c7 0x00bb2940
3 0x00b8f8bc 0x00b8f8c4 mscorlib_ni+216cf6 0x00000000
4 0x00b8f8c8 0x00b8f8d8 mscorlib_ni+22019f 0x00000000
5 0x00b8f8dc 0x00b8f8f0 mscorlib_ni+216c74 mscorlib_ni+220130
6 0x00b8f8f4 0x00b8f900 mscorwks+1b4c 0x00000000
7 0x00b8f904 0x00b8f980 mscorwks!DllUnregisterServerInternal+6195
mscorwks+1b19
8 0x00b8f984 0x00b8fab8 mscorwks!CoUninitializeEE+2e95 mscorwks!
DllUnregisterServerInternal+60f6
9 0x00b8fabc 0x00b8fad4 mscorwks!CoUninitializeEE+2ec8 mscorwks!
CoUninitializeEE+2d3b
10 0x00b8fad8 0x00b8faec mscorwks!CoUninitializeEE+2ee6 mscorwks!
CoUninitializeEE+2ea9
11 0x00b8faf0 0x00b8fcd4 mscorwks!CorExitProcess+1e4d mscorwks!
CoUninitializeEE+2ecc
12 0x00b8fcd8 0x00b8fce8 mscorwks!CoUninitializeEE+4df3 0x00000000
13 0x00b8fcec 0x00b8fd7c mscorwks!CoUninitializeEE+4d8f mscorwks!
CoUninitializeEE+4dc8
14 0x00b8fd80 0x00b8fdb8 mscorwks!CoUninitializeEE+4cb5 mscorwks!
CoUninitializeEE+4ce0
15 0x00b8fdbc 0x00b8fde0 mscorwks!CoUninitializeEE+4e41 mscorwks!
CoUninitializeEE+4c90
16 0x00b8fde4 0x00b8fdf8 mscorwks!CorExitProcess+1c1e mscorwks!
CoUninitializeEE+4e1c
17 0x00b8fdfc 0x00b8fe94 mscorwks!CorExitProcess+1cf8 mscorwks!
CorExitProcess+1c0b
18 0x00b8fe98 0x00b8ffb8 mscorwks!CreateApplicationContext+bc35 0x00000000
19 0x00b8ffbc 0x00b8ffec kernel32!GetModuleHandleA+df 0x00000000
792d6cf4 ffd0 call eax
792e019d ffd0 call eax
792d6c6f e8bc940000 call mscorlib_ni+0x220130 (792e0130)
79e71b49 ff5518 call dword ptr [ebp+18h]
79e821ac e868f9feff call mscorwks+0x1b19 (79e71b19)
79e964fc e811bcfeff call mscorwks!DllUnregisterServerInternal+0x60f6
(79e82112)
79e9652f e873feffff call mscorwks!CoUninitializeEE+0x2d3b (79e963a7)
79e9654d e8c3ffffff call mscorwks!CoUninitializeEE+0x2ea9 (79e96515)
79f3d7fe e8358df5ff call mscorwks!CoUninitializeEE+0x2ecc (79e96538)
79e9845c ff560c call dword ptr [esi+0Ch]
79e983f6 e839000000 call mscorwks!CoUninitializeEE+0x4dc8 (79e98434)
79e9831c e82b000000 call mscorwks!CoUninitializeEE+0x4ce0 (79e9834c)
79e984a8 e84ffeffff call mscorwks!CoUninitializeEE+0x4c90 (79e982fc)
79f3d5cf e8b4aef5ff call mscorwks!CoUninitializeEE+0x4e1c (79e98488)
79f3d6a9 e813ffffff call mscorwks!CorExitProcess+0x1c0b (79f3d5c1)
79f92013 ffd6 call esi
77e64826 ff5508 call dword ptr [ebp+8]
In EnfieldSMSService__PID__1764__Date__01_21_2011__Time_09_34_31AM__554__Second_Chance_Exception_C0000005.
dmp the assembly instruction at 0x00bb29d1 which does not correspond to any
known native module in the process has caused an access violation exception
(0xC0000005) when trying to read from memory location 0x00000008 on thread 10