By default, Application Verifier has these settings turned on:
Which tests will it perform in that configuration?
Each checkbox in that list can be right-clicked to see more details. In the "Verifier Stop Options", you can see the Stop number and a description. By default, there are 106 verifier stops activated (in Windows 10 21H2; the number may change with new Windows versions).
These 106 checks are:
Exceptions
650 [...] application is trying to run code from an address that is non-executable or free.
Handles
300 [...] invalid handle to system routines
301 [...] invalid TLS index to TLS system routines
302 [...] WaitForMultipleObjects with NULL as the address of the array of handles to wait for or with zero as the number of handles
303 [...] passed a NULL handle to system routines
304 [...] running code inside the DllMain function of one of the DLLs loaded in the current process and it calls WaitForSingleObject or WaitForMultipleObjects
305 [...] calling an API with a handle to an object with an incorrect object type
Heaps
001 Not used right now.
002 [...] Typically it is caused by a buffer overrun error
003 A heap created with HEAP_NO_SERIALIZE flag is not supposed to be accessed simultaneously from two threads
004 [...] HeapAlloc() or HeapReAlloc() operation the size of the block is above any reasonable value
005 Magic value of heap structure destroyed
006 [...] block gets allocated in one heap and freed in another
007 [...] block is freed twice
008 [...] corruption in the heap block
009 [...] try to destroy the default process heap
00A [...] access violation is raised in illegitimate situations
00B [...] cannot determine any particular type of corruption for the block
00C [...] cannot determine any particular type of corruption for the block
00D [...] written to after being freed
00E Freed blocks are sometimes marked non-accessible and a program touching them will access violate
00F [...] buffer overrun errors
010 [...] buffer underruns
011 [...] buffer underruns
012 [...] buffer underruns
013 [...] Typically it is caused by a buffer overrun error
014 [...] while calling GetProcessHeaps the page heap manager detects some internal inconsistencies
Leak
900 [...] owner dll of the allocation was dynamically unloaded while owning resources
901 [...] owner dll of the handle was dynamically unloaded while owning resources
902 [...] owner dll of the registry key was dynamically unloaded while owning resources
903 [...] owner dll of the virtual reservation was dynamically unloaded while owning resources
904 [...] owner dll of the SysString was dynamically unloaded while owning resources
905 [...] dll registered for power notification and was dynamically unloaded without unregistering
906 [...] owner dll of the COM allocation was dynamically unloaded while owning resources
Locks
200 [...] thread is terminated, suspended or is in a state in which it cannot hold a critical section
201 [...] global variable containing a critical section and the DLL is unloaded but the critical section has not been deleted
202 [...] heap allocation contains a critical section, the allocation is freed and the critical section has not been deleted
203 [...] critical section has been initialized more than one time
204 [...] memory containing a critical section was freed but the critical section has not been deleted
205 [...] DebugInfo field of the critical section is pointing freed memory
206 [...] owner thread ID is invalid in the current context
207 [...] recursion count field of the critical section structure is invalid
208 [...] critical section is owned by a thread if it is deleted or if the critical section is uninitialized
209 [...] critical section is released more times than the current thread acquired it
210 [...] critical section is used without being initialized or after it has been deleted
211 [...] critical section is reinitialized by the current thread
212 [...] current thread is calling VirtualFree on a memory block that contains an active critical section
213 [...] current thread is calling UnmapViewOfFile on a memory block that contains an active critical section
214 [...] calling LeaveCriticalSection but, according to the internal verifier bookkeeping, it doesn't own any critical section
215 [...] current thread tries to use a private lock that lives inside another DLL
Memory
600 [...] VirtualFree or a DLL unload with an invalid start address or size of the memory allocation
601 [...] VirtualAlloc call with an invalid start address or size of the memory allocation
602 [...] MapViewOfFile call with an invalid base address or size of the mapping
603 [...] IsBadXXXPtr call with an invalid address
604 [...] IsBadXXXPtr call for a memory allocation that is free
605 [...] IsBadXXXPtr call for a memory allocation that contains at least one GUARD_PAGE
606 [...] IsBadXXXPtr call with a NULL address
607 [...] IsBadXXXPtr call with an invalid start address or invalid size for the memory buffer to be probed
608 [...] DLL unload with an invalid start address or size of the DLL memory range
609 [...] VirtualFree for a block of memory that is actually part of the current thread's stack
60A [...] VirtualFree with an incorrect value for the FreeType parameter
60B [...] VirtualFree for an address that is already free
60C [...] VirtualFree (MEM_RELEASE) with a non-zero value for the dwSize parameter
60D [...] DllMain function is raising an exception
60E [...] thread function is raising an exception
60F [...] exception during an IsBadXXXPtr call
610 [...] VirtualFree (MEM_RESET) call with a NULL first parameter
612 [...] HeapFree, for a block of memory that is actually part of the current thread's stack
613 [...] UnmapViewOfFile, for a block of memory that is actually part of the current thread's stack
614 [...] use NULL or some other incorrect address as the address of a valid object
615 [...] use NULL or some other incorrect address as the address of a valid object
616 [...] run code from an address that is non-executable or free
617 [...] exception while initializing a buffer specified as output parameter for a Win32 or CRT API
618 [...] calling HeapSize for a heap block that is being freed
619 [...] VirtualFree (MEM_RELEASE) with an lpAddress parameter that is not the base address
61A [...] UnmapViewOfFile with an lpBaseAddress parameter that is not [correct]
61B [...] callback function in the threadpool thread is raising an exception
61C [...] run code from an address that is non-executable or free
61D [...] creating an executable heap
61E [...] allocating executable memory
SRWLock
250 [...] trying to use the SRW lock that is not initialized
251 [...] SRW lock is being re-initialized
252 [...] SRW lock is being released with a wrong release API
253 [...] SRW lock is being acquired recursively by the same thread
254 [...] thread that owns the SRW lock is exiting or being terminated
255 [...] SRW lock is being released by the thread that didn't acquire the lock
256 [...] memory address being freed contains an active SRW lock that is still in use
257 [...] DLL being unloaded contains an active SRW lock that is still in use
ThreadPool
700 [...] thread priority is changed when it's returned to threadpool
701 [...] thread affinity is changed when it's returned to threadpool
702 [...] message left as unprocessed when this threadpool thread is returned to the pool
703 [...] window is kept alive when this threadpool thread is returned to the pool.
704 [...] ExitThread is called on a threadpool thread
705 [...] call back function change the thread token to impersonate another user and forgot to reset it
706 Threadpool thread requiring a persistent thread, e.g. to access TLS
707 [...] call back function forgot to close or reset the current transaction handle
708 [...] call back function calls CoInit and CoUnInit unbalanced
709 [...] period to signal the timer is not zero when the timer is set to signal only once
70A [...] loader lock is held within the callback and is not released when the thread is returned to the threadpool
70B [...] preferred language is set within the callback and is not cleared when the thread is returned to the threadpool
70C [...] background priority is set within the callback and is not disabled when the thread is returned to the threadpool
70D [...] TerminateThread is called on a threadpool thread
TLS
350 [...] DLL that allocated a TLS index is being unloaded before freeing that TLS index
351 [...] internal verifier structures used to store the state of TLS slots for thread are corrupted
352 [...] invalid TLS index is used
Related
I have a process which does mmap into a kernel driver to map DMA packet memory into user space. When the process crashes want this memory to be part of coredump for post crash analysis.
Currently linux marks VM_IO flag for VMAs created as part of driver mmap function. In elf_core_dump() vma_dump_size() is called which checks for VM_IO flag and returns 0 if set. So these vmas are never added to coredump sections.
As per always_dump_vma() there seems probably a workaround to set vma->vm_ops
having a name function pointer which should return non-NULL when invoked.
When extending this workaround to this driver mmap function running into second problem where the vma added is filled with zeroes by kernel elf_core_dump() instead of real data. In elf_core_dump() get_dump_page() for the DMA VMA pages returns NULL as the VMA has VM_IO flag. This results in dump_skip() instead of dump_emit() being called causing the problem.
How to workaround this second problem to get non-zero contents into process coredump for this packet memory vma ? Tried searching in various forums but did not get a precise solution.
I have a .NET application running in production environment (WINDOWS XP + .NET 3.5 SP1) with a stable handle count around 2000, but in some unknown situation, its handle count will increase extremely fast and finally crash itself(over 10,000 which monitored by PerfMon tool).
I've made a memory dump from there during the increasing period (not crash yet) and imported to WinDbg, can see the overall handle summary:
0:000> !handle 0 0
7229 Handles
Type Count
None 19
Event 504
Section 6108
File 262
Port 15
Directory 3
Mutant 56
WindowStation 2
Semaphore 70
Key 97
Token 2
Process 3
Thread 75
Desktop 1
IoCompletion 9
Timer 2
KeyedEvent 1
so no surprise, the leak type is the Section, dig more:
0:000> !handle 0 ff Section
Handle 00007114
Type Section
Attributes 0
GrantedAccess 0xf0007:
Delete,ReadControl,WriteDac,WriteOwner
Query,MapWrite,MapRead
HandleCount 2
PointerCount 4
Name \BaseNamedObjects\MSCTF.MarshalInterface.FileMap.IBC.AKCHAC.CGOOBGKD
No object specific information available
Handle 00007134
Type Section
Attributes 0
GrantedAccess 0xf0007:
Delete,ReadControl,WriteDac,WriteOwner
Query,MapWrite,MapRead
HandleCount 2
PointerCount 4
Name \BaseNamedObjects\MSCTF.MarshalInterface.FileMap.IBC.GKCHAC.KCLBDGKD
No object specific information available
...
...
...
...
6108 handles of type Section
can see the BaseNamedObjects' naming convention are all MSCTF.MarshalInterface.FileMap.IBC.***.*****.
Basically I was stopped here, and could not go any further to link the information to my application.
Anyone could help?
[Edit0]
Tried several combination of GFlags command(+ust or via UI), with no luck, the dumps opened with WinDbg always see nothing via !htrace, so have to using attach process which finally I got the stack for above leaking handle:
0:033> !htrace 1758
--------------------------------------
Handle = 0x00001758 - OPEN
Thread ID = 0x00000768, Process ID = 0x00001784
0x7c809543: KERNEL32!CreateFileMappingA+0x0000006e
0x74723917: MSCTF!CCicFileMappingStatic::Create+0x00000022
0x7473fc0f: MSCTF!CicCoMarshalInterface+0x000000f8
0x747408e9: MSCTF!CStub::stub_OutParam+0x00000110
0x74742b05: MSCTF!CStubIUnknown::stub_QueryInterface+0x0000009e
0x74743e75: MSCTF!CStubITfLangBarItem::Invoke+0x00000014
0x7473fdb9: MSCTF!HandleSendReceiveMsg+0x00000171
0x7474037f: MSCTF!CicMarshalWndProc+0x00000161
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\Windows\system32\USER32.dll -
0x7e418734: USER32!GetDC+0x0000006d
0x7e418816: USER32!GetDC+0x0000014f
0x7e4189cd: USER32!GetWindowLongW+0x00000127
--------------------------------------
and then I got stuck again, the stack seems not contain any of our user code, what is the suggestion for move forward?
WinDbg isn't the ideal tool for memory leaks, especially not without preparation in advance.
There's a GFlags option (+ust) which can be enabled for a process to record the stack trace for handle allocations. If you don't have this flag enabled, you'll probably not get more info out of your dump. If you have it, use !htrace to see the stack.
You can also try UMDH (user mode dump heap), which is a free tool. Or get something like memory validator which has certainly a better usability, so it might pay off in the long run.
I'm trying to download Xcode (onto version El Capitan) and it seems to be stuck. When I run 'top', I see a process called 'storedownloadd' and the "STATE" column is alternating between sleeping, stuck,and running. The 'FAULTS' has a quickly increasing number with a plus sign after it. The 'FAULTS' column is now over 400,000 and increasing. other than 'top', I see no sign of activity of the download. Does this indicate that something is amiss? Here's a screen shot:
Processes: 203 total, 2 running, 10 stuck, 191 sleeping, 795 threads 11:48:14
Load Avg: 4.72, 3.24, 1.69 CPU usage: 56.54% user, 6.41% sys, 37.3% idle SharedLibs: 139M resident, 19M data, 20M linkedit. MemRegions: 18620 total, 880M resident, 92M private, 255M shared. PhysMem: 7812M used (922M wired), 376M unused.
VM: 564G vsize, 528M framework vsize, 0(0) swapins, 512(0) swapouts. Networks: packets: 122536/172M in, 27316/2246K out. Disks: 78844/6532M read, 240500/6746M written.
PID COMMAND %CPU TIME #TH #WQ #PORT MEM PURG CMPRS PGRP PPID STATE BOOSTS %CPU_ME %CPU_OTHRS UID FAULTS COW MSGSENT MSGRECV SYSBSD SYSMACH
354 storedownloadd 0.3 00:47.58 16 5 200 255M 0B 0B 354 1 sleeping *3[1] 155.53838 0.00000 501 412506+ 54329 359852+ 6620+ 2400843+ 1186426+
57 UserEventAgent 0.0 00:00.35 22 17 378 4524K+ 0B 0B 57 1 sleeping *0[1] 0.23093 0.00000 0 7359+ 235 15403+ 7655+ 24224+ 17770
384 Terminal 3.3 00:12.02 10 4 213 34M+ 12K 0B 384 1 sleeping *0[42] 0.11292 0.04335 501 73189+ 482 31076+ 9091+ 1138809+ 72076+
When top reports back FAULTS it's referring to "page faults", which are more specifically:
The number of major page faults that have occurred for a task. A page
fault occurs when a process attempts to read from or write to a
virtual page that is not currently present in its address space. A
major page fault is when disk access is involved in making that page
available.
If an application tries to access an address on a memory page that is not currently in physical RAM, a page fault occurs. When that happens, the virtual memory system invokes a special page-fault handler to respond to the fault immediately. The page-fault handler stops the code from executing, locates a free page of physical memory, loads the page containing the data needed from disk, updates the page table, and finally returns control to the program — which can then access the memory address normally. This process is known as paging.
Minor page faults can be common depending on the code that is attempting to execute and the current memory availability on the system, however, there are also different levels to be aware of (minor, major, invalid), which are described in more detail at the links below.
↳ Apple : About The Virtual Memory System
↳ Wikipedia : Page Fault
↳ Stackoverflow.com : page-fault
I tried to convert the kprobe as loadable kernel module.
I am able to run the samples available in samples/kprobes/ folder from
kernel tree.
If we configure kprobes in kernel(CONFIG_KPROBES), then svc_entry macro will be expanded with 64 bytes in __und_svc() handler.
Reference :
http://lxr.free-electrons.com/source/arch/arm/kernel/entry-armv.S?a=arm#L245
My aim is without touching kernel side, make kprobe as kernel module.
so kernel is compiled without enabling CONFIG_KPROBES. so svc_entry macro will be expanded with 0 in
__und_svc()
I would like to get cleared from these doubts.
If kprobe is handled undefined instruction exception(bcos kprobe
only created), then why __und_svc() is invoked. what is the role of __und_svc() handler with respect to kprobes??
If 64 bytes memory is compulsory, then how to allocate without
compiling the kernel. i.e How to do it dynamically.??
Please share your knowledge.
You may not get responses as your understanding of things is not very good and it will take some time for anyone on the linux-arm-kernel list to respond. Read kprobes.txt and study the ARM architecture in detail.
If kprobe is handled undefined instruction exception(bcos kprobe only created), then why __und_svc() is invoked. what is the role of __und_svc() handler with respect to kprobes?
On the ARM, mode 0b11011 is the undefined instruction mode. The flow when an undefined instruction happens is,
lr_und = pc of undef instruction + 4
SPSR_und = CPSR of mode where the instruction occurred.
Change mode to ARM with interrupt disabled.
PC = vector base + 4
The main vector table of step four is located at __vectors_start and this just branches to
vector_und. The code is a macro called vector_stub, which makes a descision to call either __und_svc or __und_usr. The stack is the 4/8k page that is reserved per process. It is the kernel page which contains both the task structure and the kernel stack.
kprobe works by placing undefined instructions at code addresses that you wish to probe. Ie, it involves the undefined instruction handler. This should be pretty obvious. It calls two routines, call_fpe or do_undefinstr(). You are interested in the 2nd case, which gets the opcode and calls call_undef_hook(). Add a hook with register_undef_hook(); which you can see arch_init_kprobes(). The main callback kprobe_handler is called with a struct pt_regs *regs, which happens to be the extra memory reserved in __und_svc. Notice for instance, kretprobe_trampoline(), which is playing tricks with the stack that it is currently executing with.
If 64 bytes memory is compulsory, then how to allocate without compiling the kernel. i.e How to do it dynamically.?
No it is not. You can use a different mechanism, but you may have to modify the kprobes code. Most likely you will have to limit functionality. It is also possible to completely re-write the stack frame and reserve the extra 64bytes after the fact. It is not an allocation as in kmalloc(). It is just adding/subtracting a number from the supervisor stack pointer. I would guess that the code re-writes the return address from the undefined handler to execute in the context (ISR, bottom half/thread IRQ, work_queue, kernel task) of the kprobed address. But there are probably additional issues you haven't yet encountered. If arch_init_kprobes() is never called, then you can just always do the reservation in __und_svc; it just eats 64 bytes of stack which will make it more likely that the kernel stack will overflow. Ie, change,
__und_svc:
# Always reserve 64 bytes, even if kprobe is not active.
svc_entry 64
arch_init_kprobes() is what actually installs the feature.
Our product consumes a lot of windows resources, such as socket handles, memory, threads and so on. Usually there are 700-900 active threads, but in some cases product can rapidly create new threads and do some work, and close it.
I came across with crash memory dump of our product. With ~* windbg command I can see 817 active threads, but when I run !handle command it prints me these summary:
Type Count
None 15
Event 2603
Section 13
File 705
Directory 4
Mutant 32
WindowStation 2
Semaphore 789
Key 208
Process 1
Thread 5766
Desktop 1
IoCompletion 308
Timer 276
KeyedEvent 1
TpWorkerFactory 48
So, actually process holds 5766 threads. So, my question, When Windows actually frees handles for process? Is it possible some kind of delay, or cashing? Can someone explain this behavior?
I don't think that we have handle leaks, but we have weird behavior in legacy part of system with rapidly creating and closing threads for small tasks. Also I would like to point, that we unlikely run more than 1000 threads simultaneously, I am pretty sure about this.
Thanks.
When you say So, actually process holds 5766 threads., what you really mean is that the process holds 5766 thread handles.
Even though a thread may no longer be running, whether that is the result of a call to ExitThread()/TerminateThread() or returning from the ThreadProc, any handles to that thread will remain valid. This makes it possible to do things like call GetExitCodeThread() on the handle of a thread that has finished its work.
Unfortunately, that means that you have to remember to call CloseHandle() instead of just letting it leak. The MSDN example on Creating Threads covers this to some extent.
Another thing that I will note is that somewhere not too far above 1000 running threads, you are likely to exhaust the amount of virtual address space available to a 32bit process since each thread by default reserves 1MB of address space for its stack.