i was searching how Windows memory works with X86
I learnt something but i am in dilemma is it true or not.
I did not found any docs on the internet relationship between VAD and GDT
Windows OS uses VAD ( Virtual Address Descriptors ) to allocate address from LDT or GDT array.
As i read , each VAD entry is just a "page table"
so :
For processes each page table is LDT entry
For Kernel each page table is GDT entry
NTOSKRNL.EXE when OS boots inits VADs
by calling RtlInitializeGenericTable
This function makes a binary tree , each node contains free LDT or GDT entry.
this binary tree root is stored inside GDT itself.
When we allocate a memory from user mode malloc , binary tree subtract one LDT node from LDT pool and VAD tree root.
My second question:
Windows can run number of processes is the limit of GDT entries
x86 can have 16bit GDT entries so we can add 65536 LDT entries
each process can have 1 LDT entry.
So GDT can contains up to 60K entries.
That means Windows can run max ~60k process
is this true ?
Related
I did some experiments about memory analysis.
I have some problems..
almost Directory Table Base can divide 4k(4096) i know.
But my process in windows 10 (1909) have 0x14695e002 DTB.
So that can't divide 4k. 2 ramians.
Why my windows have that value??
The dirBase / Directory Table base is the value of the CR3 register for the current process. As you may know the CR3 is the base register which (indirectly) points to the base of the PML4 (or PDPT) table and is used when switching between process, which basically switches their entire physical memory.
Base CR3
As you may have seen in the Intel manual the 4 lower bits of the CR3 should be ignored by the CPU (Format of the CR3 register with 4-Level Paging):
4-level paging
Now if you look closely at the at the Intel Manual (Chapter 4.5; 4-level Paging).
A logical processor uses 4-level paging if CR0.PG = 1, CR4.PAE = 1, and IA32_EFER.LME = 1
Respectively: Paging; Physical Address Extension; Long Mode Enable.
Use of CR3 with 4-level paging depends on whether process context identifiers (PCIDs) have been enabled by setting CR4.PCIDE.
CR4.PCIDE
CR4.PCIDE is documented in the Intel Manual (Chapter 2.5 Control Registers):
CR4.PCIDE
PCID-Enable Bit (bit 17 of CR4) — Enables process-context identifiers (PCIDs) when set. See Section 4.10.1, “Process-Context Identifiers (PCIDs)”. Can be set only in IA-32e mode (if IA32_EFER.LMA = 1).
So when CR4.PCIDE is set, the 12 (0:11) lower bits of CR3 are used as PCID, that is, a "Process-Context Identifier" (bits 12 to M-1, where M is usually 48, are used for the physical address for the base of the PML4 table).
PCIDs
PCIDs are documented in the Intel Manuel (Chapter 4.10.1; Process-Context Identifiers (PCIDs)):
Process-context identifiers (PCIDs) are a facility by which a logical processor may cache information for multiple linear-address spaces. The processor may retain cached information when software switches to a different linear address space with a different PCID.
And a little bit further in the same chapter:
When a logical processor creates entries in the TLBs [...] and paging-structure caches [...], it associates those entries with the current PCID.
So basically PCIDs (as far as I understand them) are a way to selectively control how the TLB and paging structure caches are preserved or flushed when a context switch happens.
Some of the instruction that operate on cacheability control (such as CLFLUSH, CLFLUSHOPT, CLWB, INVD, WBINVD, INVLPG, INVPCID, and memory instructions with a non-temporal hint) will check the PCID to either flush everything that concerns a precise PCID or flush only a part of the cache (such as the TLB) and keep everything in relation to a given PCID.
For example the INVPLG instruction:
The INVLPG instruction normally flushes TLB entries only for the specified page; however, in some cases, it may flush more entries, even the entire TLB. The instruction invalidates TLB entries associated with the current PCID and may or may not do so for TLB entries associated with other PCIDs.
The INVPCID specifically uses the PCIDs:
Invalidates mappings in the translation lookaside buffers (TLBs) and paging-structure caches based on process-context identifier (PCID)
Why it is always 2 (as far as I can see, it's always 2 for every processes in the system) on Windows, I don't know.
I have read multiple articles on this topic including below but things are still hazy to me:
http://elinux.org/Tims_Notes_on_ARM_memory_allocation
ARM Linux kernel page table
Linux kernel ARM Translation table base (TTB0 and TTB1)
ARM hardware has 4096 entries of 4 byte each in L1 translation table. each entry translates a 1MB region in memory. At second level it has 256 entries of 4 bytes each. And each of second level entry translates a 4KB page in memory.
So according to this any virtual address has to be divided into 12-8-12 to map to above scheme.
But on 32 bit ARM linux side this division is 11-9-12. Where L1 translation table consists of 2048 entries where each entry is 8 bytes. Here two 4 byte entries are clubbed together and the pointed second level translation tables are laid out one after the other in memory, so that at second level instead of 256 there are 512 entries. Additionally since Linux memory management expects various flags non native to ARM we define 512 more entries for linux page table(one for each 2nd level HW page table).
Now the question is Linux does not enforce PGD/PMD/PTE size (however it enforces page size to be 4K. Thus PAGE_SHIFT is set to 12), then why do we select 11-9-12 layout(i.e. 11 bits for PGD and 9 bits for HW PTE).
Is it just to make sure that 512HW +512Linux PTE are aligned to a Page boundary ?
If someone could explain the logic behind this division in detail would be great....
As you say, in the ARM short-descriptor format each second-level page table is 1KB in size. Even with the associated shadow page table that only makes 2KB, meaning 50% of every page allocated for second-level tables would be entirely wasted.
Linux just pretends that the section size is 2MB, rather than the actual 1MB of the hardware, by allocating first-level entries in pairs, so that the corresponding pair of second-level tables can be kept together in a single page, avoid that wastage, and keep the management of page table memory really simple.
The ARM Linux and dirty bits should have all the answers. Mainly, the PTE tables have extra info to emulate bits resulting in the layout you observe.
I think a misconception is the memory an L2 table occupies versus what it maps. You must allocate physical memory for an L2 table and having it symmetric (4K size) make it the same as all pages. Now this 4k page could be four ARM MMU L2 page tables. However, we need some additional information to emulate dirty, young and accessed bits that the Linux generic MMU code requires. So the layout of the Linux L2 (PTE directory) is,
Linux PTE [n]
Linux PTE [n+1]
ARM PTE [n]
ARM PTE [n+1]
At the L1 level each entry is paired (n/n+1) so that it points to item 3 and 4 above. The pgtable-2level.h file has detailed comments on the layout (which should be correct for your version of Linux).
See: Tim's notes on ARM MM
Page table entry (PTE) descriptor in Linux kernel for ARM
My question is in handle_mm_fault function or any S/W page table walk.
pgd = pgd_offset(mm,address);
pud = pud_offset (pgd,address);
pmd = pmd_offset (pud,address);
pte = pte_offset_map(pmd,address);
Will the final computed pte be the ARM or LINUX version?
In armv7 supports 2 levels of page table; At the first level 4096 entries of each 4 bytes has an address for the second level. The second level has 256 entries of each 4 bytes. Linux has tweaked the page table to have 2048 entries of each 8 bytes in other words having two pointers to second level of page table having 512 entries placed contiguously. Linux PTE stored below these 512 ARM PTE.
So I understand that page table walk through in S/W will leads to ARM PTE only, but that is not correct Linux always operates on Linux PTEs?
Please tell me where I am wrong?
I Got the answer.
I understood that, all these macro will leads to the Linux PTE only , since the L2 level page table of 4kb size and from 0th offset the linux pte0 starts and 1024th offset linux pte1 starts. while calculating pte_index it mask the lower 12 bits of PMD value thus PMD will always point to the start of page and there are Linux PTE stored.
I Got the answer. I understood that, all these macro will leads to the Linux PTE only , since the L2 level page table of 4kb size and from 0th offset the linux pte0 starts and 1024th offset linux pte1 starts. while calculating pte_index it mask the lower 12 bits of PMD value thus PMD will always point to the start of page and there are Linux PTE stored.
When I study OS,I find a concept Logical Memory.So Why there is a need for a Logical Memory?How does a CPU generate Logical Memory?The output of "&ptr" operator is Logical or physical Address?Is Logical Memory and Virtual Memory same?
If you're talking about C's and C++'s sizeof, it returns a size and never an address. And the CPU does not generate any memory.
On x86 CPUs there are several layers in address calculations and translations. x86 programs operate with logical addresses comprised of two items: a segment selector (this one isn't always specified explicitly in instructions and may come from the cs, ds, ss or es segment register) and an offset.
The segment selector is then translated into the segment base address (either directly (multiplied by 16 in the real address mode and in the virtual 8086 mode of the CPU) or by using a special segment descriptor table (global or local, GDT or LDT, in the protected mode of the CPU), the selector is used as an index into the descriptor table, from where the base address is pulled).
Then the sum segment base address + offset forms a linear address (AKA virtual address).
If the CPU is in the real address mode, that's the final, physical address.
If the CPU is in the protected mode (or virtual 8086), that linear/virtual address can be further translated into the physical address by means of page tables (if page translation is enabled, of course, otherwise, it's the final physical address as well).
Physical memory is your RAM or ROM (or flash). Virtual memory is physical memory extended by the space of disk storage (could be flash as well as we now have SSDs).
You really need to read up on this. You seem to have no idea.
I have the following page table of process1 :
Assuming that the paging system works with 16bit addresses and page size is 4k
And I want to convert the logical address 16000 to a physical address .
I'm a little bit new in this topic so go easy on me :
Partial solution : The address 16000 fits cell number 3 in the page table , so I guess
that I need to work with that cell and its stored frame - 2 .
How can I find the offset and the physical address now ?
Thanks
In your case process 1 currently can access upto 4 * 4k bytes of virtual memory.
Generally a process can access upto 4gb of virtual memory(depending on the implementation).
Now the table you have given maps virtual memory to the actual physical address(on RAM).With each entry of page table mapping 4k of memory from virtual to physical space.
So the physical address where the address 16000 corresponds to the 3rd entry of page table which is mapped to the physical address starting from 8192(3*4096) till 12288 (8192+4096).
16000 mod 4096 = 3712(offset).
At an offset of 3172 bytes in the virtual page 2 i.e at an offset of 3172 in physical page 3 ( at address 8192) you find the data corresponding to address 16000.
All these mappings are done by the MMU(memory management unit) for every address access a process makes.
Good link to understand this concept is here.
Cheers :)