I am trying to investigate the behaviour of the Windows Kernel with respect to Memory Mapped Files / Virtual Memory. Specifically I am interested in determining how frequently the contents of a memory mapped file are flushed (by Windows) to disk and what criterion Windows uses for deciding it is time to do so.
I have done a bit of research online and, apart from the MSDN which deals more with the 'hows and whys' rather than detailing the internal workings, there doesn't appear to be much information. If anyone can point me to any articles or has looked into this before and has some insight I would welcome it.
Thanks.
I suggest reading Microsoft Windows Internals by Mark Russinovich and David Solomon, in fifth edition also with Alex Ionescu.
Memory mapped files are controlled by the modified page writer in the kernel - they're flushed whenever the memory that backs the page needs to be re-used (so it can happen under memory pressure).
The system also keeps track of the number of dirty pages and writes them after a threshold is reached.
You can find more information about in this article about Windows memory management changes for Windows Vista - it doesn't directly answer your question about how the memory manager treats modified pages but you can infer some of the details based on the information in the article.
The internal workings are an implementation detail and subject to change. They're not documented anywhere because Microsoft doesn't want you relying on the details.
I've got to ask: Why do you care? If you have hard real-time requirements, Windows might not be the OS for you...
I would suggest that you obtain a copy of
Microsoft® Windows® Internals
Fourth Edition: Microsoft Windows Server 2003 Windows XP and Windows 2000
There are pdfs floating around in the cyber cloud.
This is the book done by the "sysinternals" crowd who have written some very good windows utils over the years.
http://www.microsoft.com/technet/sysinternals
You will have to read the chapters on
Memory Management
Storage Management
Cache Manager
File Systems
They don't directly address your concern but if you manage to read, digest and grasp the very techical paragraphs you can work it out yourself. However if you can predict why Windows does what it does "You are a better man than most".
Happy brain freeze reading that lot.
Related
Is there any way to figure out which application is using up all the desktop heap memory?
For an explanation of 'desktop heap' see this MSDN blog.
EDIT: If you don't know what "desktop heap memory" is please don't answer.
EDIT2: if you don't know what "desktop heap memory" is don't vote to close the question.
Disclaimer: I work for Microsoft, and these are my personal thoughts and experiences which may or may not reflect current or future Microsoft policies/procedures/etc.
I know this is old, but since I stumbled across it I wanted to provide some feedback/answers.
Desktop Heap Monitor does not work on Vista+. DHeapMon depended on the ability to patch kernel binaries to track desktop heap- this behavior is blocked, which is why it doesn't work.
To monitor desktop heap yourself, all you can do is look at user objects (unfortunately).
To get a complete answer, you can open a support ticket with Microsoft and with a kernel or complete dump (NOT a process dump!) and they can inspect the heap and provide you some information. It is extracted from the dump file via a debugger extension, which I wrote.
Limitations:
Only works from kernel mode. Even though a desktop heap mapping exists in user mode, the extension does not support it. Even if I added support for user mode debugging of desktop heap, it would be limited as some of the related data is only in kernel mode. Changes on recent builds of Windows 10 might restrict it further, so I have not invested time to add user mode support.
It can't measure what isn't in the dump. In other words, if the desktop heap is missing pages it will do its best but it will obviously lose accuracy.
Does not support Windows 10 build 1809 or Server 2019, yet, due to changes to the heap manager.
Happy coding!
According to this blog post you should look for the user objects.
To find the bad application you can configure the USER Objects column within Process Explorer. Then sort descending by user objects. User Objects are all GUI objects like Windows, Toolbars, Icons, Cursors ... The process with the highest values is very likely the bad one.
Direct link to download: DHeapMon from microsoft.
If you want to do it within your program, C++ can use WMI to find the culprit.
I'm using a commandline tool to do some processing on a file. The thing is that this file should not be stored on disk (security reasons). So I was wondering whether it's possible in Windows to use a part of memory as a virtual file that is accessible bu the commandline tool as if it was a real physical file.
Yes, it's possible with things referred to as "ramdisks" usually. What's the best ramdisk for Windows? over at superuser.com has some links.
Have you written the command line tool yourself? If so, you can simply allocate a section of memory to your program and use it in your processing. There's little reason to trick the app into thinking it's using a file on a physical disk. The specifics on how to do so depend on what language your app is written in.
If not, you'll need to create a RAM disk and tell the program to use that. Using a RAM disk on Windows requires third-party software; a comprehensive list of options is available here on Super User.
Note, though, neither using a RAM disk nor storing all of your data in memory will make it more secure. The information stored in RAM is just as accessible to prying eyes and malicious applications as data that is saved on the hard disk. Probably more so than data that has been deleted from the hard disk.
If you need a ready to use application, there are several ramdisk applications (including free ones) on the market, and then your question here is offtopic. If you need to do this in code, than one of our virtual storage products (SolFS, CallbackDisk, Callback File System) will work, and Callback File System has a sample project that stores files in memory.
If you're using .NET, you might look into MemoryStream.
Note Cody Gray's answer though, which is only too true insofar as having something in memory does not guarantee that it can't be compromised. Though opinions differ on this subject. Most people would argue that writing to disk is even less secure, especially in the age of wear-levelling where controlling what is deleted and what is not is practically impossible.
RAM has its own disadvantages, but on the positive side, what's gone is gone :-)
I need to find the underlying disk capacity (total size) of an unmapped network share in windows (in Win7, Vista, XP, Server 2008), given a UNC path (e.g. given something like "\\share_1\subdir").
I've looked all over the web for several days and seem to find no answer to this issue. I would appreciate any leads. Thanks in advance for your time!
I would have given up by now, if it weren't for the ability to find the underlying free space of unmapped network shares, using the "GetDiskFreeSpaceEx()" Win32 function. I imagine that disk capacity is stored in a similar fashion to free space, hence retrieving it would be very similar (hence I'm somewhat infuriated with MS for not making the functionality obvious, or myself for not being able to find it thus far!)
Regards,
vivri
You are on the right track. GetDiskFreeSpaceEx will also display capacity, you just have to call the correct members.
See this Microsoft support link on how to do it.
Keep in mind that GetDiskFreeSpaceEx may only retrieve the free disk space by user. For instance, Windows Explorer uses GetDiskFreeSpaceEx also and it may not report the actual free physical disk space, but rather the user's logged in quota.
I couldn't find a suitable title for this. I'm going to express my query with examples.
Consider following softwares:
Process explorer from sysinternals (an advanced task manager)
Resource Manager : resmon.exe (lists each and every fine detail about resource usage about each process).
For me these softwares seems like miracles. I wonder how these are even made. C'mon how a user process can know such fine details about other processes? Who tells this software, what processes are running and what all resources are utilized? Which dlls are used? etc..
Does windows operating system give these software that information? I mean though (obviously the most lower level api) WIN32API. Are there some functions,which on calling return these values
abstractly say:
GetAllRunningProcesses()
GetMemoryUsedByProcess(Process* proc)
etc..
Other similar applications are
network Packet Capture software. How does it get information about all those packets? It clearly sits just infront of the NIC card. How is it possible?
Anti-virus: It scans memory for viruses. Intercepts other processes. Acts like a sandbox for the user application space. How? How??
If its WIN32API. I swear, I'm going to master it.
I don't want to create a multi-threaded application. I want to get information about other multithreaded applications.
I don't want to create a program which communicates using sockets. I want to learn how to learn how to capture all communication packets.
I actually want to work at the lower level. But I don't know, what should I learn. Please guide me in proper direction.
This is really a pretty open-ended question. For things like a list of running processes, look up "PSAPI" or "Toolhelp32". For memory information about a particular process, you can use VirtualQuery.
Capturing network packets is normally done by installing a device driver. If you look, you should be able to find a fair amount about how to write device drivers, though don't expect to create wonders overnight, and do expect to crash your machine a few times in the process (device drivers run in kernel mode, so it's easy for a mistake to crash the machine hard).
I can't say as much with any certainty about anti-virus, because I've never tried to write one. My immediate guess would be that their primary technique is API hooking. There's probably more to it than that, but offhand I've never spent enough time looking at them to know what.
Mark Russinovich's classic, Windows Internals, is the go-to book if you want to get deep in this kind of stuff. I notice that the just-released 5th edition includes Vista. Here's a sample chapter to peek at.
If you like Process Explorer, this is the guy who wrote that, and there are lots of examples using it in the book.
Plus, at 1232 hardcover pages, you can use it to press your clothes.
Seriously, I've trawled MSDN and only got half answers - what do the columns on the Task Manager mean? Why can't I calculate the VM Usage by enumerating threads, modules, heaps &c.? How can I be sure I am accurately reporting to clients of my memory manager how much address space is left? Are their myriad collisions in the memory glossary namespace?
An online resource would be most useful in the short term, although books would be acceptable in the medium term.
Try the book "Windows Internals" by Mark Russinovich and I think some other guy too. It's pretty good on getting down to the nitty gritty.
Mark Russinovich has written the excellent book Windows Internals. A new edition that covers the Vista and Server 2008 operating systems is currently in the works with David Solomon, so you may want to pre-order that if your questions are about the new Windows operating systems instead of the old ones.
Here is a quick article on Windows Memory Management, which goes into sufficient depth to interpret what you're actually seeing in Task Manager or Process Explorer.