Is there any reason that the Valid-Bit in a page table would ever be turned off (set to invalid)? Also when working with the dirty bit, I know that the dirty bit is supposed to be set whenever there is a write request on the page table entry in question. However, why is the dirty bit useful?
Is there any reason that the Valid-Bit in a page table would ever be turned off (set to invalid)?
When a page table is initialized because one page table entry (pte) is created, the other ptes in that page table need to be marked invalid so they don't get used.
If the page gets kicked/swapped out from memory, its pte needs to be marked invalid so that the OS can intercept the page fault upon any future read/writes and swap the correct memory back in.
Also when working with the dirty bit, I know that the dirty bit is
supposed to be set whenever there is a write request on the page table
entry in question. However, why is the dirty bit useful?
The dirty bit allows you to detect which pages have been written to in memory only (not yet propagated back to disk). If the OS wants to evict a dirty page, it sees that the dirty bit is set and now knows that it should write the changes back to disk before evicting the page. Without the dirty bit, 1) either the OS would need to compare every bit of the page to its backing disk-page to check if it was dirty before evicting it, or 2) it would have to adopt a burdensome "always-write-back" policy for every page eviction, even if the page were a clean, unwritten page.
Related
I feel very clear on what happens with a segmentation fault and a major page fault, but I'm a bit more curious on the subtleties of minor page faults, and maybe an example would be with dynamically linked libraries. Wikipedia says, for instance:
If the page is loaded in memory at the time the fault is generated, but is not marked in the memory management unit as being loaded in memory, then it is called a minor or soft page fault. The page fault handler in the operating system merely needs to make the entry for that page in the memory management unit point to the page in memory and indicate that the page is loaded in memory; it does not need to read the page into memory. This could happen if the memory is shared by different programs and the page is already brought into memory for other programs.
The line, "The page fault handler in the operating system merely needs to make the entry for that page in the memory management unit point to the page in memory" confuses me. Each process has its own page table. So if I try to map in, say, libc, what's the process that the kernel goes through to figure out that it's already been mapped? How does it know that another process is using it or that there's already a frame associated with it? Does this happen with the page cache? I was reading a bit about it here, but I think some clarification would be nice on the steps that occur in the kernel to identify and resolve a minor page fault would be helpful.
Edit: It looks like a radix tree is used to keep track? Although I'm not quite sure I'm understanding this correctly.
At first, the kernel has no idea if the page is in memory or not. Presumably, the process does have a handle open to the file however, so the kernel performs an operation through the kernel-side file descriptor entry. This involves calling into the filesystem which, of course, knows what pages of the file are resident in memory since it's the code that would load the page were one needed.
From a website about TLB: (https://www.bottomupcs.com/virtual_memory_hardware.xhtml#other-page-related-faults):
For the following parts I highlighted:
1: Is the format of TLB entries the same as PTE(page table entries)? And it's not clear
the "the page" in the page can be marked as mean TLB entry or PTE?
2: For "the pages" in go through all the pages, are they TLB entries or PTEs?
3: Why it's moved out of not "moved into"?
4: Is that the order is (1)set the two bits (2)put into TLB, or the converse?
There are two other important faults that the TLB can generally generate which help to mange accessed and dirty pages. Each page generally contains an attribute in the form of a single bit which flags if the page has been accessed or is dirty.
An accessed page is simply any page that has been accessed. 1When a page translation is initially loaded into the TLB the page can be marked as having been accessed (else why were you loading it in?[19])
2The operating system can periodically go through all the pages and clear the accessed bit to get an idea of what pages are currently in use. When system memory becomes full and it comes time for the operating system to choose pages to be swapped out to disk, obviously those pages whose accessed bit has not been reset are the best candidates for removal, because they have not been used the longest.
A dirty page is one that has data written to it, and so does not match any data already on disk. For example, if a page is loaded in from swap and then written to by a process, 3before it can be moved out of swap it needs to have its on disk copy updated. A page that is clean has had no changes, so we do not need the overhead of copying the page back to disk.
Both are similar in that they help the operating system to manage pages. The general concept is that a page has two extra bits; the dirty bit and the accessed bit. 4When the page is put into the TLB, these bits are set to indicate that the CPU should raise a fault.
When a process tries to reference memory, the hardware does the usual translation process. However, it also does an extra check to see if the accessed flag is not set. If so, it raises a fault to the operating system, which should set the bit and allow the process to continue. Similarly if the hardware detects that it is writing to a page that does not have the dirty bit set, it will raise a fault for the operating system to mark the page as dirty.
I suggest ignoring that link as a source. It is highly confusing. I do not see any mention of a specific implementation but it is clearly describing one.
In any rationally designed processor, the TLB is completely transparent to programmers (even systems programmers). It is entirely a piece of hardware.
1: Is the format of TLB entries the same as PTE(page table entries)?
A programmer never sees TLB entries. They could have the same format as PTEs. They might not.
2: For "the pages" in go through all the pages, are they TLB entries or PTEs?
Programmers have no access to TLB entries. They have to refer to PTEs.
3: Why it's moved out of not "moved into"?
Probably but there seems to be a lot of confusion in your link.
4: Is that the order is (1)set the two bits (2)put into TLB, or the converse?
This is describing some specific, unnamed implementation. Most processors have a dirty bit but not all have an accessed bit.
I'm trying to write a toy working set estimator, by keeping track of page faults over a period of time. Whenever a page is faulted in, I want to record that it was touched. The scheme breaks down when I try to keep track of accesses to already-present pages. If a page is read from or written to without triggering a fault, I have no way of tracking the access.
So then, I want to be able to cause a "lightweight" fault to occur on a page access. I've heard of some method at some point, but I didn't understand why it worked so it didn't stick in my mind. Dirty bit maybe?
You can use mprotect with PROT_NONE ("Page cannot be accessed"). Then any access to the given page will cause a fault.
The usual way to do this is to simply clear the "present" bit for the page, while leaving the page in memory and the necessary kernel data structures in place so that the kernel knows this.
However, depending on the architecture in question you may have better options - for example, on x86 there is an "Accessed" flag (bit 5 in the PTE) that is set whenever the PTE is used in a linear address translation. You can simply clear this bit whenever you like, and the hardware will set it to record that the page was touched.
Using either of these methods you will need to clear the cached translation for that page out of the TLB - on x86 you can use the INVLPG instruction.
My understanding is that, in general, there is one page table per process.
My question is then: what happens in case of a page fault of the running process, if the OS has to send a page frame of a non-running process back to disk. The page table of the non-running process has to be updated too, or else when it is running again, its page table will not be correct. However in all the texts I read this is not mentioned. Am I misunderstanding something?
Each process has a page table, but the page table is for that process and not owned by that process, but owned by the kernel (this is a bit weird wording, but I hope you get what I mean). So when a non running process is paged out to disk the page table itself is still 'running', and updated. I don't think the page table itself is ever swapped out to disk, as this would cause very weird problems.
I have a memory mapped file, and a page in a view which is currently committed. I would like to decommit it. MapViewOfFile tells me I cannot use VirtualFree on file mapped pages. Is there some other way to do it?
You cannot decommit it, but what you really want is not decommitting it ...
What you really want is to release the page from a memory. This can be done by using VirtualUnlock. See VirtualUnlock Remarks:
Calling VirtualUnlock on a range of memory that is not locked releases the pages from the process's working set.
Note: As documented, the function will return FALSE (the page was not locked) and GetLastError will return ERROR_NOT_LOCKED.
This is described in Guillermo Prandi's question CreateFileMapping, MapViewOfFile, how to avoid holding up the system memory.
Remarks: I think you can view it this way: decommitting a mapped page is nonsense - pages is commited whenever it is backed by a physical storage, be it a memory or a file. File mapped page cannot be decommitted in this sense, as it will be always backed by the file.
However, the code in the question mentioned is measuring the memory footprint, but what it measures is not representative, as the fact the page is removed from the process working set does not necessarily mean it is no longer present in a memory.
I have performed a different experiment, measuring how long it takes to read a byte from a memory mapped page. After unlocking the page or unmapping the view and closing the mapping handle the access was still fast.
For the access to be slow (i.e. to really discard the page from the memory) it was necessary to unmap the view and close BOTH memory mapping handle and file handle (the last was surprising to me, as I expected unmapping the view and closing the mapping handle will be enough).
It is still possible system will take VirtualUnlocked as a hint and it will discard the pages sooner, once it needs to discard something, but this is something I have to think about yet how to prove.