Seriously, I've trawled MSDN and only got half answers - what do the columns on the Task Manager mean? Why can't I calculate the VM Usage by enumerating threads, modules, heaps &c.? How can I be sure I am accurately reporting to clients of my memory manager how much address space is left? Are their myriad collisions in the memory glossary namespace?
An online resource would be most useful in the short term, although books would be acceptable in the medium term.
Try the book "Windows Internals" by Mark Russinovich and I think some other guy too. It's pretty good on getting down to the nitty gritty.
Mark Russinovich has written the excellent book Windows Internals. A new edition that covers the Vista and Server 2008 operating systems is currently in the works with David Solomon, so you may want to pre-order that if your questions are about the new Windows operating systems instead of the old ones.
Here is a quick article on Windows Memory Management, which goes into sufficient depth to interpret what you're actually seeing in Task Manager or Process Explorer.
Related
Is there any way to figure out which application is using up all the desktop heap memory?
For an explanation of 'desktop heap' see this MSDN blog.
EDIT: If you don't know what "desktop heap memory" is please don't answer.
EDIT2: if you don't know what "desktop heap memory" is don't vote to close the question.
Disclaimer: I work for Microsoft, and these are my personal thoughts and experiences which may or may not reflect current or future Microsoft policies/procedures/etc.
I know this is old, but since I stumbled across it I wanted to provide some feedback/answers.
Desktop Heap Monitor does not work on Vista+. DHeapMon depended on the ability to patch kernel binaries to track desktop heap- this behavior is blocked, which is why it doesn't work.
To monitor desktop heap yourself, all you can do is look at user objects (unfortunately).
To get a complete answer, you can open a support ticket with Microsoft and with a kernel or complete dump (NOT a process dump!) and they can inspect the heap and provide you some information. It is extracted from the dump file via a debugger extension, which I wrote.
Limitations:
Only works from kernel mode. Even though a desktop heap mapping exists in user mode, the extension does not support it. Even if I added support for user mode debugging of desktop heap, it would be limited as some of the related data is only in kernel mode. Changes on recent builds of Windows 10 might restrict it further, so I have not invested time to add user mode support.
It can't measure what isn't in the dump. In other words, if the desktop heap is missing pages it will do its best but it will obviously lose accuracy.
Does not support Windows 10 build 1809 or Server 2019, yet, due to changes to the heap manager.
Happy coding!
According to this blog post you should look for the user objects.
To find the bad application you can configure the USER Objects column within Process Explorer. Then sort descending by user objects. User Objects are all GUI objects like Windows, Toolbars, Icons, Cursors ... The process with the highest values is very likely the bad one.
Direct link to download: DHeapMon from microsoft.
If you want to do it within your program, C++ can use WMI to find the culprit.
I want to test the performance of a filesystem under different conditions.
Specifically I want to test the performance of Windows virtual machines without compression and with compression both on "normal harddisk" and on USB-disk as it would be interesting to see exactly what the difference is.
What I need is a program that can test different aspects of filesystem (random access, sequential read/write, etc) and make pretty graphs that go well with my blog. Preferrably the application should be automated so I can add it to startup, this way the timing is the same for each run and I can repeat the runs for verification.
I can post a link to the results here when I get around to testing it. Right now its just in the planning phase.
Iometer is the I/O measurement tool. And it's free. From the website:
Iometer is an I/O subsystem
measurement and characterization tool
for single and clustered systems. It
was originally developed by the Intel
Corporation and announced at the Intel
Developers Forum (IDF) on February 17,
1998 - since then it got wide spread
within the industry.
Meanwhile Intel has discontinued to
work on Iometer and it was given to
the Open Source Development Lab
(OSDL). In November 2001, a project
was registered at SourceForge.net and
an initial drop was provided. Since
the relaunch in February 2003, the
project is driven by an international
group of individuals who are
continuesly improving, porting and
extend the product.
The tool (Iometer and Dynamo
executable) is distributed under the
terms of the Intel Open Source
License. The iomtr_kstat kernel module
as well as other future independent
components are distributed under the
terms of the GNU Public License.
You said you'd like pretty graphs for your blog. In my use of IOMeter, I've never seen it produce a graph. However, it is possible that I overlooked an existing feature.
Alternatively, (from the look of its website) iozone might give you graphs:
http://www.iozone.org/
Yet, it could be that iozone only collected the data used to create those graphs shown on its web site.
Regardless, this is still another option for I/O Benchmarking.
Additional server oriented disk benchmarks:
Diskspd
fio
vdbench
I am currently in an university operating system class and we are working on the windows kernel, more precisely WRK, the windows research kernel, for our projects. WRK is based off of win2k3 server.
I am however having a real hard time dredging up resources to help learn the basics of OS development, Windows kernel development and just generally getting around the Windows API.
We are using the book Microsoft Internals by Russinovich but I was wondering if any of you had some great resources to recommend to me, whether book, online guides or some old class notes. Thanks!
What specifically are you looking for? Online resources? For that, OSROnline is one of the better websites. Alot of kernel development knowledge is found in the MS and the OSR Mailing lists, that's another place to check that might be better than Stack overflow.
Specifically books, there is the Programming WDM,Developing drivers with KMDF and Advance Windows Debugging. The last specifically will not teach you so much about the kernel and more how to navigate inside it, something you will do quite often if you are writing drivers or researching parts of it.
In order to write drivers, the easiest way is probably to take Windows Driver samples and hack at them, stare the results with windbg and learn more.
microsoft kernel dev? that's just weird. what university are you at?
one of the most interesting things about kernels, in my opinion, is the scheduler algorithms. I'd recommend you check that out.
I can't imagine where you'd start looking for windows stuff though. I did it with the linux kernel and there's a LOT of resources (of course).
http://oreilly.com/catalog/linuxkernel/chapter/ch10.html
The third edition of Tanenbaum's Modern Operating Systems has a chapter devoted to the Vista kernel. I haven't looked into that chapter (I only read the Linux one), but as far as big-picture stuff, it's fantastic. I'm not sure what level of detail you're looking for, but that might be a good resource to check out.
I am trying to investigate the behaviour of the Windows Kernel with respect to Memory Mapped Files / Virtual Memory. Specifically I am interested in determining how frequently the contents of a memory mapped file are flushed (by Windows) to disk and what criterion Windows uses for deciding it is time to do so.
I have done a bit of research online and, apart from the MSDN which deals more with the 'hows and whys' rather than detailing the internal workings, there doesn't appear to be much information. If anyone can point me to any articles or has looked into this before and has some insight I would welcome it.
Thanks.
I suggest reading Microsoft Windows Internals by Mark Russinovich and David Solomon, in fifth edition also with Alex Ionescu.
Memory mapped files are controlled by the modified page writer in the kernel - they're flushed whenever the memory that backs the page needs to be re-used (so it can happen under memory pressure).
The system also keeps track of the number of dirty pages and writes them after a threshold is reached.
You can find more information about in this article about Windows memory management changes for Windows Vista - it doesn't directly answer your question about how the memory manager treats modified pages but you can infer some of the details based on the information in the article.
The internal workings are an implementation detail and subject to change. They're not documented anywhere because Microsoft doesn't want you relying on the details.
I've got to ask: Why do you care? If you have hard real-time requirements, Windows might not be the OS for you...
I would suggest that you obtain a copy of
Microsoft® Windows® Internals
Fourth Edition: Microsoft Windows Server 2003 Windows XP and Windows 2000
There are pdfs floating around in the cyber cloud.
This is the book done by the "sysinternals" crowd who have written some very good windows utils over the years.
http://www.microsoft.com/technet/sysinternals
You will have to read the chapters on
Memory Management
Storage Management
Cache Manager
File Systems
They don't directly address your concern but if you manage to read, digest and grasp the very techical paragraphs you can work it out yourself. However if you can predict why Windows does what it does "You are a better man than most".
Happy brain freeze reading that lot.
I'm looking for an explanation or good free online resources about the organization of memory and memory management in Windows systems.
You can buy this (old) book, (the actual Windows architecture hasn't really changed much in 10 years) for a penny (used):
Inside Windows NT
Here's the up-to-date version:
MS Windows Internals
There's also Mark's Blog
As far as ONLINE resources go, I've found it to be pretty sparse, though :(
Check out this DotNet Rocks podcast on the Windows memory model
In MSDN, some information is located here: http://msdn.microsoft.com/en-us/library/aa366525(VS.85).aspx
Windows Internals is the canonical reference for mm. It's far better than anything else you'll find on the subject, as Landy Wang doesn't write too much :)