as I read here the virtual address space of a 32 bit Windows application has 2GB of storage (from 0x00000000-0x7FFFFFFF). The other 2GB are reserved for the system address space.
However, I found a pointer in a 32bit program (using Cheat Engine) which is pointing to an address which isn't in range of the virutal address space. The addresses in my last exploration were 0x301DDC3C -> 0x87F56190 like you can see in the picture:
(The expansion in the first line means a dereference of the pointer 0x301DDC3C, in the next line you can see what's in the dereference location 0x87F56190 in RAM)
After dereferencing the pointer there are pointers back into the process virtual address space.
How is it possible that a user mode application has a valid pointer into system address space?
Does this mean the pointer in location 0x301DDC3C is pointing to an location in the system address space? And so the process I'm examining is using kernel mode stuff?
from Memory and Address Space Limits
Limits on memory and address space vary by platform, operating system, and by whether the IMAGE_FILE_LARGE_ADDRESS_AWARE flag in the IMAGE_FILE_HEADER.Characteristics. IMAGE_FILE_LARGE_ADDRESS_AWARE (The application can handle addresses larger than 2 GB) is set or cleared by using the /LARGEADDRESSAWARE linker option.
by default IMAGE_FILE_LARGE_ADDRESS_AWARE cleared for 32-bit PE and set for 64-bit PE, but we can overwrite default:
so 32-bit process with set IMAGE_FILE_LARGE_ADDRESS_AWARE flag - up to 4Gb memory is avaible.
really of course [0, 0x800000000000) (win8.1 +) or [0, 0x80000000000) (before win 8.1) memory space is avaible for user mode in x64 windows. but system artificially restrict this by reserve big range of memory (this allocation is protected and can not be free)
for 32-bit process this reservation begin from 7FFF0000 or FFFE0000 and up to 64-bit ntdll.dll. very interesting that in 64-bit process, where IMAGE_FILE_LARGE_ADDRESS_AWARE cleared - also was such reserved memory space begin from 0x80000000. also interesting that in this case kernel32.dll is loaded at another address compare usual 64-bit process. so base of kernel32.dll not the same in general in all 64-bit processes. but ntdll.dll loaded at the same address in all processes anyway.
usual memory allocations on x64 windows:
32 bit process, IMAGE_FILE_LARGE_ADDRESS_AWARE cleared (default)
32 bit process, IMAGE_FILE_LARGE_ADDRESS_AWARE set
64 bit process, IMAGE_FILE_LARGE_ADDRESS_AWARE cleared
64 bit process, IMAGE_FILE_LARGE_ADDRESS_AWARE set (default)
ALL of the addresses you see are virtual addresses, of the process (not "physical" addresses). A user-space process may use pointers that happen to come from "system space", but that does NOT mean a process can freely access kernel resources , nor does it mean that these pointers necessarily map to physical addresses.
Here is another Microsoft link, that might help clarify:
Virtual Address Space
When a processor reads or writes to a memory location, it uses a
virtual address. As part of the read or write operation, the processor
translates the virtual address to a physical address.
...
The range of
virtual addresses that is available to a process is called the virtual
address space for the process. Each user-mode process has its own
private virtual address space. For a 32-bit process, the virtual
address space is usually the 2-gigabyte range 0x00000000 through
0x7FFFFFFF.
...
Processes like Notepad.exe and MyApp.exe run in user
mode. Core operating system components and many drivers run in the
more privileged kernel mode. For more information about processor
modes, see User mode and kernel mode. Each user-mode process has its
own private virtual address space, but all code that runs in kernel
mode shares a single virtual address space called system space. The
virtual address space for a user-mode process is called user space.
...
In 32-bit Windows, the total available virtual address space is
2^32 bytes (4 gigabytes). Usually the lower 2 gigabytes are used for
user space, and the upper 2 gigabytes are used for system space.
...
Code running in user mode has access to user space but does not have
access to system space. This restriction prevents user-mode code from
reading or altering protected operating system data structures. Code
running in kernel mode has access to both user space and system space.
That is, code running in kernel mode has access to system space and
the virtual address space of the current user-mode process.
...
It's also worthwhile to note the difference betwee kernel mode and user mode:
User mode and kernel mode
When you start a user-mode application, Windows creates a process for
the application. The process provides the application with a private
virtual address space and a private handle table. Because an
application's virtual address space is private, one application cannot
alter data that belongs to another application. Each application runs
in isolation, and if an application crashes, the crash is limited to
that one application. Other applications and the operating system are
not affected by the crash.
...
In addition to being private, the virtual address space of a user-mode application is limited. A processor running in user mode
cannot access virtual addresses that are reserved for the operating
system. Limiting the virtual address space of a user-mode application
prevents the application from altering, and possibly damaging,
critical operating system data.
...
Related
If there is a 32 bit system (assume Windows), the virtual address space is 4GB. So CPu can generate any address between this range. Then shoudn't a process also be able to address anywhere in this range?
It is said that each process has its own private virtual address space.Then How does the system facilitate this?
In other words the CPU generates a 32 bit address, and that gets translated into physical address. Now how does CPU know that a specific process has to address only a specific part of the virtual address space(its private virtual address space).
Suppose a process addresses an address out of its private virtual address space, what happens?
A program has to call VirtualAlloc() on Windows to tell the operating system that it wants to use a chunk of virtual memory. Often called indirectly as a result of allocating memory from a heap or loading a DLL.
The operating system, in turn, sets up the page mapping tables that the CPU uses to translate a virtual address as used in the program to a physical RAM address as output on its address bus pins. One of three unusual things can happen whenever the CPU reads or writes data or executes code at a virtual memory address:
if there is no entry in the page mapping tables then the CPU raises a general protection fault trap. The operating system verifies that the address is invalid and terminates the program
if the page is not mapped to RAM yet then the CPU raises a page fault trap. The operating system finds a page of RAM that's unused, swapping out a used page if necessary. And ensures the content is valid, loading it from a file or the paging file if necessary. And updates the table entry so it now has the physical address of the RAM page. Execution resumes as normal
the CPU verifies that access to the page is allowed. A write to a page that is marked as read-only or an execute of a instruction in the page that's marked as no-execute generates a general protection fault trap. The operating system terminates the program.
Every process has its own set of page mapping tables, ensuring that one process cannot access the RAM pages that are used by another. Unless sharing is specifically requested, common for pages of code loaded from an executable file and memory mapped files. A context switch loads the CR2 register, the CPU register that contains the address of the page mapping table.
So there is no scenario where a process can ever address memory outside of its private virtual address space, the lack of a matching paging table entry ensures that this terminates the program.
The whole 4 GB address space is available to the process (although typically the upper half is reserved for kernel data), and the MMU maps parts of it to physical memory. The process cannot go "out" of its address space (all the 4 GB of it are allowed to be used), but if some part of it hasn't been mapped to physical memory a hardware exception is raised.
The address space is said to be private since the operating system changes the settings of the MMU at task switch, so every process sees a different independent memory layout (although parts of the address space can be shared with other processes).
In Windows the high memory of every process (0x80000000 or 0xc0000000)
Is reserved for kernel code, user code cannot access these regions of memory, if it tries so an access violation exception will be thrown.
I wish to know how is the kernel space protected ?
Is it via memory segmentations or via paging ?
I would like to hear a technical explanation.
Thanks a lot,
Michael.
Assuming you are talking about x86 and x64 architectures.
Memory protection is achieved using the paging system. Each page table entry on an x86/x64 CPU has a bit to indicate whether it is a user or supervisor page. Accesses to supervisor pages are only permitted for code running with CPL<3, whereas accesses to non supervisor pages are possible regardless of CPL.
CPL is the "Current Privilege Level" which is sometimes referred to as Ring. Windows only uses two rings, although the CPU implements 4. Ring 0 is the CPU mode in which what Windows refers to as "kernel mode" runs. Ring 3 is the CPU mode in which "User mode" runs. Since code running at CPL=3 cannot access supervisor pages, this is how memory protection is implemented.
The answer for ARM is likely to be similar, but different.
That's an easy one and doesn't require talking about rings and kernel behavior. Accessing virtual memory at a particular address requires that address to be mapped, the operating system has to allocate a memory page for that address. The low-level winapi function that does that is VirtualAlloc(). Which takes an optional address, first argument. The OS will simply fail a request for an unmappable address. Otherwise the exact same mechanism that prevents you from mapping any address in the lowest 64KB of the address space.
This might be a silly question but it just popped up in my mind. All the text about process address space and virtual memory layout mentions that the process address space has
space reserved for kernel. For e.g. on 32 bit systems the process address space is 4GB of which 1 GB is reserved for kernel in Linux (Might be different on other OS).
I am just wondering why kernel is said to be in the process address space when a process cannot address the kernel directly. Why don't we say that the kernel has a separate address space than a process and why can't we have a different page table for kernel itself which is separate from the page tables of the processes?
When the process makes a system call, we don't need to switch the page tables (from process address space page table to kernel address space page table) for servicing the system call (which should be done only in kernel mode). This is said to be that the kernel is running in the process context.
Some kernel events which won't run in process context will load the page tables only for kernel.
Got it ?
If I understand correctly, a memory adderss in system space is accesible only from kernel mode. Does it mean when components mapped in system space are executed the processor must be swicthed to kernel mode?
For ex: the virtual memory manager is a frequently used component and is mapped in system space. Whenever the VMM runs in the context of user process (lets say it translated an address), does the processor must be swicthed to kernel mode?
Thanks,
Suresh.
Typically, there's 2 parts involved.The MMU(Memory manage unit) which is a hardware component that does the translation from virtual addresses to physical addresses. And the operating system VM subsystem.
The operating system part needs to run in privileged mode (a.k.a. kernel mode) and will set up/change the mapping in the MMU based on the the user space needs.
E.g. to request more (virtual) memory, or map a file into memory, a transition to kernel mode is needed and the VM subsystem can change the mapping of the process.
Around this there's often a ton of tricks to be made - e-g. map the whole address space of the kernel into the user process virtual space, but change its access so the process can't use that memory - this means whenever you transit to kernel mode you don't need to reload the mapping for the kernel.
Taking your example of the virtual memory manager, it never actually runs in user space. To allocate memory, user mode applications make calls to the Win32 API (NTDLL.DLL as one example) to routines such as VirtualAlloc.
With regards to address translation, here's a summary of how it works (based on the content from Windows Internals 5th Edition).
The VMM uses page tables which the CPU uses to translate virtual addresses to physical addresses. The page tables live in the system space. Each table contains many PTEs (page table entries) which stores the physical address to which a virtual address is mapped. I won't go into too much detail here, but the point is that all of the VMM's work is performed in system space and not in user space.
As for context switching - when a thread running in user space needs to run in the system space, then a context switch will occur. Since the memory manager lives in system space, it's threads never need to make a context switch, since it already lives in the system space.
Apologies for the simplistic explanation, this is quite a complicated topic of discussion in depth. I would highly recommend that you pick up a copy of Windows Internals as this sounds like it would come in handy for you.
I'm writing a diagnostic app which needs to log what the user has set as his user-mode space a.k.a. user-mode virtual address space
a.k.a. the /3GB switch in WinXP or the increaseuserva switch in bcdedit on Vista/Win7.
Either of C++ or C++/CLI will do.
Any ideas ?
GlobalMemoryStatusEx will give you a MEMORYSTATUSEX struct with ullTotalVirtual:
The size of the user-mode portion of the virtual address space of the calling process, in bytes. This value depends on the type of process, the type of processor, and the configuration of the operating system. For example, this value is approximately 2 GB for most 32-bit processes on an x86 processor and approximately 3 GB for 32-bit processes that are large address aware running on a system with 4-gigabyte tuning enabled.
Note that you'd have to mark your EXE as LARGEADDRESSAWARE in order to see 3GB in your process.
I think there's another function that also returns this info (no - not GlobalMemoryStatus which is deprecated) along with processor info - but I can't recall it ATM.