Windows 8 added the PrefetchVirtualMemory which is a great help with avoiding hard page faults when reading a lot of sequential data with memory mapped files.
Windows 8.1/10 also added DiscardVirtualMemory which would be great as to avoid reading in pages that will be overwritten. However, when invoking DiscardVirtualMemory on a memory address owned by a Memory Mapping the thread hangs and the function call never completes.
Is DiscardVirtualMemory supported with Memory Mapped files. If so, does one need to do something special (i.e. undocumented) to make it work?
Related
As I understand the creation of processes, every process has it's own space in RAM for it's heap, data, etc, which is allocated upon its creation. Many processes can share their data and storage space in some ways. But since terminating a process would erase its allocated memory(so also its caches), I was wondering if it is possible that many (similar) processes share a cache in memory that is not allocated to any specific process, so that it can be used even when these processes are terminated and other ones are created.
This is a theoretical question from a student perspective, so I am merely interested in the general sence of an operating system, without adding more functionality to them to achieve it.
For example I think of a webserver that uses only single-threaded processes (maybe due to lack of multi-threading support), so that most of the processes created do similar jobs, like retrieving a certain page.
There are a least four ways what you describe can occur.
First, the system address space is shared by all processes. The Operating system can save data there that survives the death of a process.
Second, processes can map logical pages to the same physical page frame. The termination of one process does not cause the page frame to be deallocated to the other processes.
Third, some operating systems have support for writable shared libraries.
Fourth, memory mapped files.
There are probably others as well.
I think so, when a process is terminated the RAM clears it. However your right as things such as webpages will be stored in the Cache for when there re-called. For example -
You open Google and then go to another tab and close the open Google page, when you next go to Google it loads faster.
However, what I think your saying is if the Entire program E.G - Google Chrome or Safari - is closed, does the webpage you just had open stay in the cache? No, when the program is closed all its relative data is also terminated in order to fully close the program.
I guess this page has some info on it -
https://www.wikipedia.org/wiki/Shared_memory
I was trying to understand following:
I know that page tables are built for translation between virtual memory and physical memory by virtual memory manager at some point. Since there are many processes running on a system, even though only process active at a time, I was wondering whether page tables for inactive process are moved to page file at any point of time? Given the fact that lower 2 GB area is reserved for windows, it would make sense that windows would keep page tables for all processes on the system. Although it would make sense as well that they are moved to page file if the current process is switched?
Same goes for the writable (data) pages. Will windows keep all the data pages for all the process in memory or move them to page file at some point. On my machine, task manager says 1.5 GB RAM is being utilized out of 3 GB and 1.5 is system cache in performance tab so my understanding is data stays in physical memory for all applications. But would there be a time when it needs to moved to paging file?
I was wondering whether page tables for inactive process are moved to page file at any point of time?
Yes, page tables are pageable.
Will windows keep all the data pages for all the process in memory or move them to page file at some point.
As far as the Windows paging policy is concerned, there's two kinds of memory: pageable and non-pageable. It doesn't really matter which process it belongs to or even if it belongs to the O/S itself, if it's pageable then it's subject to being paged out. So, yes, Windows will page out process data pages if necessary.
I suggest reading the memory management chapter in the Windows Internals book, it should cover all of this.
-scott
You are actually asking two questions here.
What's the paging policy regarding the page tables.
What's the paging policy for "writable data" pages (i.e. virtual memory with R/W permissions).
First I'll correct you a little.
Given the fact that lower 2 GB area is reserved for windows, it would
make sense that windows would keep page tables for all processes on
the system
To be exact it's the upper 2GB that are reserved to windows, more correctly - may be accessed in the kernel mode only by Windows kernel and drivers.
Now, this may surprise you, but the kernel memory may be pagable too! So technically it's not important at all which portion of the 32-bit address space is visible in the user/kernel mode. It's not related to paging.
Another correction: virtual memory may be in physical memory and saved to the page file. There's a common belief that the OS frees physical storage by on-demand saving the pages to the page file. Wrong.
Actually Windows saves memory pages to the page file before they need to be freed. In fact it dumps all the memory pages to the page file (besides of those that are related to other files, such as mapped sections) in background. There are two reasons for this:
During high load the OS will free memory pages quicker (since they're already saved)
In the kernel mode paging is not always possible. Drivers that run on high IRQL (i.e. serve the most time-critical events) may not access physical storage drivers, hence paging is not possible.
So, the answers to your questions are:
Don't know for sure, but it depends on the OS implementation details. I see no reasons why per-process page table may not be paged-out. It's needed during the context switch and modifying process virtual memory. Both situations don't belong to the time-critical events.
Definitely "writable data" memory pages are saved to the page file. Are they removed from the physical memory? On-demand only, during the system load, in the least-recent-used order.
Are you meant to protect against simultanously reads/writes to file mapped memory that is open by multiple processes?
For example if a string in the memory is "hello" and one process writes "hi..." over it, am I correct to say that another process that reads at the same time may get an intermittant value like "hi.lo"?
Basically what I am asking is how do people protect again these sorts of things. Are you meant to use semaphores? Do these work across processes?
Yes, if you need to protect against multiple writers or avoid reading partial updates then a shared Mutex / Semaphore used by each process would work to control access to the shared data.
There is some sample code which does this at the bottom of this MSDN article: Memory-Mapped Files in .NET 4.0
CreateFileMapping and MapViewOfFile, and then we do I/O with a function like memcpy.
Just use Read/WriteFile.
Is the first one faster than second?
I don't understand that.
Why is it faster?
If we use a section object, then we can get more cache benefits from VMM or Cache Manager?
File memory mapping is faster when page out occurs, as the file itself is used as paging storage.
If the memory in the memory mapped file is not changed there is no need to flush the the page to the paging file as the data is in the file already and windows can reread the page from disk. .EXE and .DLL files are loaded using this mechanism and thus are their own page storage.
If the memory in the memory mapped file is written, then page out is the same as if the paging file had been used. Faster possibly as the same place on disk is used (subject to NTFS optimisations).
The plain APIs consume page file backed memory to hold the contents of the file while in memory.
On slightly different perspective both APIs are optimised as memory mapped files are may actually be used the Read/Write File APIs under the hood so you get a micro-optimisation as you're using a lower abstraction
Both mechanisms will employ the VMM/Cache manager.
Use of ReadFile/WriteFile involves several extra memory block copying operations, so it will be slower, than use of MMFs. Another question is how much slower it will be - this is what you need to measure yourself.
I have a memory mapped file, and a page in a view which is currently committed. I would like to decommit it. MapViewOfFile tells me I cannot use VirtualFree on file mapped pages. Is there some other way to do it?
You cannot decommit it, but what you really want is not decommitting it ...
What you really want is to release the page from a memory. This can be done by using VirtualUnlock. See VirtualUnlock Remarks:
Calling VirtualUnlock on a range of memory that is not locked releases the pages from the process's working set.
Note: As documented, the function will return FALSE (the page was not locked) and GetLastError will return ERROR_NOT_LOCKED.
This is described in Guillermo Prandi's question CreateFileMapping, MapViewOfFile, how to avoid holding up the system memory.
Remarks: I think you can view it this way: decommitting a mapped page is nonsense - pages is commited whenever it is backed by a physical storage, be it a memory or a file. File mapped page cannot be decommitted in this sense, as it will be always backed by the file.
However, the code in the question mentioned is measuring the memory footprint, but what it measures is not representative, as the fact the page is removed from the process working set does not necessarily mean it is no longer present in a memory.
I have performed a different experiment, measuring how long it takes to read a byte from a memory mapped page. After unlocking the page or unmapping the view and closing the mapping handle the access was still fast.
For the access to be slow (i.e. to really discard the page from the memory) it was necessary to unmap the view and close BOTH memory mapping handle and file handle (the last was surprising to me, as I expected unmapping the view and closing the mapping handle will be enough).
It is still possible system will take VirtualUnlocked as a hint and it will discard the pages sooner, once it needs to discard something, but this is something I have to think about yet how to prove.