Is there a way in Windows/Linux or any other operating system to know at instruction-level, if a memory access generated a Page Fault? The code I'm imagining about would look something like this:
Buffer *buffer = new Buffer();
...Do something with the buffer...
if(thisProcess.generatedPageFault) {
...Do something...
}
...Do something else with the buffer...
if(thisProcess.generatedPageFault) {
...Do something...
}
For linux the closest thing you'll get is reading /proc/self/stat and parse out the
no. of pagefaults before and after your calls - the format is described here: http://linux.die.net/man/5/proc
Keep in mind, reading/parsing that file could itself cause pagefaults - atleast the 1. time you do it, of if you allocate memory to read it (like calling fopen)
Accordingly to the Intel documentation for the x86 processor - a page fault is Interrupt 14. The kernel at a low level would have an interrupt handler set up to trap that page fault. When that occurs, the kernel's interrupt handler handles the situation accordingly.
Now, since this is at the nuts and bolts level, and residing in ring 0 code, I would not think you can actually monitor that...you may get around that by creating a driver to simply watch for a page fault (again, dependent on the OS privileges and internal data structures occupied by the kernel), and pass the information back out to user-land space... I doubt if that would be easily exposable...
Hope this helps,
Best regards,
Tom.
Related
Background
I have an ZynqMP system which has four Cortex-A53 cores (PS) along with FPGA logic (PL). They transfer data via AXI bus.
I've placed some Xilinx AXI Quad SPI in my design. Linux which runs on PS successfully probes them, and starts a daemons which periodically (333 Hz) ask MCUs on SPIs to reply their data chunk (~ up to around 500 bytes, split in every 64 bytes.)
They works nicely for a while (median 50 minutes) but suddenly the readl_relaxed() in SPI driver causes Synchronous External Abort which leads an Kernel Panic. It seems to be an AXI's error reply according to ARM TRM, and might be recoverable because it's "synchronous" which means the registers are not corrupted (in my understanding.)
After some search I found the do_sea() func that handles SEA and also found that there's no chance to recover from it according to the implementation.
I want the AXI error to be handled like: discard the read, return SIGBUS and lead the process to be killed, etc.
Of course I'm debugging the Abort and finding why it occurs but at present I have no clue.
Question
So my questions are:
Why SEAs are not recoverable in Linux arm64 implementation?
If I can "handle" or "ignore" it, how do I modify Linux kernel code (I know it's stupid but I'd like to know if there's a way.)
What can reply error in Quad SPI IP? The readl_relaxed I mentioned above reads Rx data FIFO.
1) I’ve never ventured down this path, but it looks to me like they are recoverable if the inf->fn returns 0; which means that ghes_notify_sea() must return 0; thus one of the SEA error sources successfully reported an error.
2) I think you need a bit more info. I would start by changing
drivers/acpi/apei/ghes.c:732
from:
rc = ghes_read_estatus(ghes, 0);
to:
rc = ghes_read_estatus(ghes, 1);
which should get you a bit more information when the error happens.
Armed with that information, you need to find out if you have a malfunctioning handler, or a missing one. Either way, this is the place to address it.
3) You are dealing with an ACPI implementation. There are 155 kloc in the kernel plus unknown quantity in the firmware and hardware. The kernel code doesn’t appear to handle whichever condition you are running into. First you need to determine which of these suspects is involved and what interactions are failing before you can dig out the root cause.
Happy Digging!
I have managed to create a virtual IOPCIDevice which attaches to IOResources and basically does nothing. I'm able to get existing drivers to register and match to it.
However when it comes to IO handling, I have some trouble. IO access by functions (e.g. configRead, ioRead, configWrite, ioWrite) that are described in IOPCIDevice class can be handled by my own code. But drivers that use memory mapping and IODMACommand are the problem.
There seems to be two things that I need to manage: IODeviceMemory(described in the IOPCIDevice) and DMA transfer.
How could I create a IODeviceMemory that ultimately points to memory/RAM, so that when driver tries to communicate to PCI device, it ultimately does nothing or just moves the data to RAM, so my userspace client can handle this data and act as an emulated PCI device?
And then could DMA commands be directed also to my userspace client without interfering to existing drivers' source code that use IODMACommand.
Thanks!
Trapping memory accesses
So in theory, to achieve what you want, you would need to allocate a memory region, set its protection bits to read-only (or possibly neither read nor write if a read in the device you're simulating has side effects), and then trap any writes into your own handler function where you'd then simulate device register writes.
As far as I'm aware, you can do this sort of thing in macOS userspace, using Mach exception handling. You'd need to set things up that page protection fault exceptions from the process you're controlling get sent to a Mach port you control. In that port's message handler, you'd:
check where the access was going to
if it's the device memory, you'd suspend all the threads of the process
switch the thread where the write is coming from to single-step, temporarily allow writes to the memory region
resume the writer thread
trap the single-step message. Your "device memory" now contains the written value.
Perform your "device's" side effects.
Turn off single-step in the writer thread.
Resume all threads.
As I said, I believe this can be done in user space processes. It's not easy, and you can cobble together the Mach calls you need to use from various obscure examples across the web. I got something similar working once, but can't seem to find that code anymore, sorry.
… in the kernel
Now, the other problem is you're trying to do this in the kernel. I'm not aware of any public KPIs that let you do anything like what I've described above. You could start looking for hacks in the following places:
You can quite easily make IOMemoryDescriptors backed by system memory. Don't worry about the IODeviceMemory terminology: these are just IOMemoryDescriptor objects; the IODeviceMemory class is a lie. Trapping accesses is another matter entirely. In principle, you can find out what virtual memory mappings of a particular MD exist using the "reference" flag to the createMappingInTask() function, and then call the redirect() method on the returned IOMemoryMap with a NULL backing memory argument. Unfortunately, this will merely suspend any thread attempting to access the mapping. You don't get a callback when this happens.
You could dig into the guts of the Mach VM memory subsystem, which mostly lives in the osfmk/vm/ directory of the xnu source. Perhaps there's a way to set custom fault handlers for a VM region there. You're probably going to have to get dirty with private kernel APIs though.
Why?
Finally, why are you trying to do this? Take a step back: What is it you're ultimately trying to do with this? It doesn't seem like simulating a PCI device in this way is an end to itself, so is this really the only way to do what greater goal you're ultimately trying to achieve? See: XY problem
I am having some issues with my virtualHBA driver on Windows Server 2016. A ran the HLK crashdump support test. 3 times out of 10 the test passed. In those 3 failing tests, the crashdump hangs at 0% while taking Complete dump, or Kernel dump or minidump.
By kernel debugging my code, I found that the call to ExAllocatePoolWithTag() for buffer allocation never actually returns.
Below is the statement which never returns.
pDeviceExtension->pcmdbuf=(struct mycmdrsp *)ExAllocatePoolWithTag(NonPagedPoolCacheAligned,pcmdqSignalSize,((ULONG)'TA1'));
I searched on the web regarding this. However, all of the found pages are focusing on this function returning NULL which in my case never returns.
Any help on how to move forward would be highly appreciated.
Thanks in advance.
You can't allocate memory in crash dump mode. You're running at HIGH_LEVEL with interrupts disabled and so you're calling this API at the wrong IRQL.
The typical solution for a hardware adapter is to set the RequestedDumpBufferSize in the PORT_CONFIGURATION_INFORMATION structure during the normal HwFindAdapter call. Then when you're called again in crash dump mode you use the CrashDumpRegion field to get your dump buffer allocation. You then need to write your own "crash dump mode only" allocator to allocate buffers out of this memory region.
It's a huge pain, especially given that it's difficult/impossible to know how much memory you're ultimately going to need. I usually calculate some minimal configuration overhead (i.e. 1 channel, 8 I/O requests at a time, etc.) and then add in a registry configurable slush. The only benefit is that the environment is stripped down so you don't need to be in your all singing, all dancing configuration.
This is actually a theoretical question about memory management. Since different operating systems implement things differently, I'll have to relieve my thirst for knowledge asking how things work in only one of them :( Preferably the open source and widely used one: Linux.
Here is the list of things I know in the whole puzzle:
malloc() is user space. libc is responsible for the syscall job (calling brk/sbrk/mmap...). It manages to get big chunks of memory, described by ranges of virtual addresses. The library slices these chunks and manages to respond the user application requests.
I know what brk/sbrk syscalls do. I know what 'program break' means. These calls basically push the program break offset. And this is how libc gets its virtual memory chunks.
Now that user application has a new virtual address to manipulate, it simply writes some value to it. Like: *allocated_integer = 5;. Ok. Now, what? If brk/sbrk only updates offsets in the process' entry in the process table, or whatever, how the physical memory is actually allocated?
I know about virtual memory, page tables, page faults, etc. But I wanna know exactly how these things are related to this situation that I depicted. For example: is the process' page table modified? How? When? A page fault occurs? When? Why? With what purpose? When is this 'buddy algorithm' called, and this free_area data structure accessed? (http://www.tldp.org/LDP/tlk/mm/memory.html, section 3.4.1 Page Allocation)
Well, after finally finding an excellent guide (http://duartes.org/gustavo/blog/post/how-the-kernel-manages-your-memory/) and some hours digging the Linux kernel, I found the answers...
Indeed, brk only pushes the virtual memory area.
When the user application hits *allocated_integer = 5;, a page fault occurs.
The page fault routine will search for the virtual memory area responsible for the address and then call the page table handler.
The page table handler goes through each level (2 levels in x86 and 4 levels in x86_64), allocating entries if they're not present (2nd, 3rd and 4th), and then finally calls the real handler.
The real handler actually calls the function responsible for allocating page frames.
Note: This is not a problem i'm experiencing, but it is something i'd
like to understand (just because i
want to be a better person, and to
further the horizon of human
understanding).
In the bonus chapter of Raymond Chen's book,
Raymond gives the example of a bug in a sound card driver:
The original function, called at
hardware interrupt time, looks like
this in the DDK:
void FAR PASCAL midiCallback(NPPORTALLOC pPortAlloc, WORD msg,
DWORD dwParam1, DWORD dwParm2) {
if (pPostAlloc->dwCallback)
DriverCallBack(pPortalloc->dwCallback, HIWORD(pPortalloc->dwFlags),
pPortalloc->hMidi, msg, dwParam1, dwParam2);
}
Their version of the function looked
like this:
void FAR PASCAL midiCallback(NPPORTALLOC pPortAlloc, WORD msg,
DWORD dwParam1, DWORD dwParm2) {
char szBuf[80];
if (pPostAlloc->dwCallback) {
wsprintf(szBuf, " Dc(hMidi=%X,wMsg=%X)", pPortalloc->hMidi, msg);
#ifdef DEBUG
OutputDebugString(szBuf);
#endif
DriverCallBack(pPortalloc->dwCallback, HIWORD(pPortalloc->dwFlags),
pPortalloc->hMidi, msg, dwParam1, dwParam2);
}
}
Not only is there leftover debug stuff in retail code, but it is
calling a noninterrupt- safe function
at hardware interrupt time. If the
wsprintf function ever gets
discarded, the system will take a
segment-not-present fault inside a
hardware interrupt, which leads to a
pretty quick death.
Now if i'm looking at that code i wouldn't have guessed that a call to the library function wsprintf would be a problem. What happens if my driver code needs to make use of the Win32 API?
What is a segment fault? i understand the concept of a page-fault: the code i need is sitting on a page that has been swapped out to the hard-drive, and will need to get back from the hard drive before code execution can continue. What is a segment fault when we're inside a device-driver's interrupt?
Is page-faults the protected mode equivalent of a segment-fault? How does one avoid segment faults? Does Windows ever swap out device driver code? How would i stop "wsprintf from being discarded"? What would cause wsprintf to be "discarded"? What is "discarded"? What is the virtue of discarding? When it something undiscarded
Why is calling an API call from inside a driver bad, and how would one work around it?
A segmentation fault normally refers to an invalid memory access. In most modern operating systems the mechanism which generates seg-faults is also used to provide the demand paging mechanism. What they tend to do is "swap" pages of memory out to disc and mark them as invalid, the next time an instruction accesses that bit of memory the kernel recognises that it isn't really an error and will page in memory.
Windows cannot handle page-faults in certain contexts, one of them being in an interrupt. That is just the way it is designed. For example imagine you get a page fault in the code which reads memory pages data from the disk drive, how could it possible handle such an occurrance? So they define certain limitations on what modes of operation are allowed to page and what are not. If you cause a page fault in an interrupt the kernel will force a BSOD.
What you are supposed to do in an interrupt context if you need to do something which might need paging is to queue what is called a Deferred Procedure Call (DPC) in the interrupt handler. The DPC is then executed at DPC level (something you will see mentioned if you read some of the descriptions of DDK functions). DPC level can page and so you can use any function you need.
As for driver stuff, you can mark some of your code as non-pageable and you can allocate non-paged-pool which is memory you can access without causing page-faults. wsprintf could be paged out because no-one has used it and the kernel reclaims the memory.