I want to monitore swap space usage on windows 2003 server. If the usage is over 80% for 10 minutes, an alarm will be generated. There are lot of tools for RAM, but how about swap usage? How do I simulate that condition and do the test?
Use the built-in performance counters. You can fetch them via WMI/Win32_Perf:
http://msdn.microsoft.com/en-us/library/aa394270%28v=VS.85%29.aspx
or the raw Performance counter/registry interfaces:
http://msdn.microsoft.com/en-us/library/aa373083%28v=VS.85%29.aspx
To force the page file to be used. Start Commiting Memmory. Use the VirtualAlloc api call:
LPVOID WINAPI VirtualAlloc(
__in_opt LPVOID lpAddress,
__in SIZE_T dwSize,
__in DWORD flAllocationType,
__in DWORD flProtect
);
and set flAllocationType to MEM_COMMIT (0x1000), this should start memory being used. Once memory is suffcient exhausted, then the page file should be automatically employed. I suspect you'll have to start measuring usage and then determine heuristically as to when %usage you require happens.
To monitor it read the performance counters. The paging file set has a %usage counter you can read. Start here on how to consume them. All you need is to create a windows service that reads the info and then rings the appropriate alarms.
.Net : https://learn.microsoft.com/en-us/archive/blogs/bclteam/how-to-read-performance-counters-ryan-byington
C++ : http://msdn.microsoft.com/en-us/library/aa373219(v=VS.85).aspx or http://msdn.microsoft.com/en-us/library/aa373214(v=VS.85).aspx
Related
I have a large binary file (about 1 GB) that I want to process sequentially. I'm using a memory mapped file to access the file. Is there a way to tell Windows to swap in the whole file as soon as possible? Currently, it looks as if windows is only loading a single page each time a page fault occurs which results in very slow processing.
I'm really surprised nobody's mentioned PrefetchVirtualMemory:
BOOL WINAPI PrefetchVirtualMemory(
_In_ HANDLE hProcess,
_In_ ULONG_PTR NumberOfEntries,
_In_ PWIN32_MEMORY_RANGE_ENTRY VirtualAddresses,
_In_ ULONG Flags
);
Here are some other links:
http://msdn.microsoft.com/en-us/library/ms810613.aspx
Performance of Win32 memory mapped files vs. CRT fopen/fread
http://duartes.org/gustavo/blog/post/page-cache-the-affair-between-memory-and-files
Personally, I have no idea whether any of this is a Good Idea or not for your application. But sure - try it, and see if it helps!
IMHO ..
When you wish to force a chunk of the file to be paged in to memory, call ReadFile using the file handle that you used to create the file mapping.
I am trying to share memory between user space and kernel space in windows xp.I want to write into the memory in user level and reading it from the kernel driver program.Is there any available in-build function?
User mode side: There are a couple of ways.
1) The most common / flexible way is to use DeviceIOControl to communicate between user mode and kernel mode, including passing memory.
It's been a long time (six years? XP timeframe) since I wrote my last kernel driver, so this is an overview of the process not exact code. However, your user mode program should be able to obtain a handle to your 'device' or an open instance of your driver working on something by using CreateFile and specifying its name, such as \\.\YourNameHere for example. Use that handle to communicate with it (the first parameter to DeviceIOControl.)
The four parameters you will be interested in are:
__in_opt LPVOID lpInBuffer,
__in DWORD nInBufferSize,
__out_opt LPVOID lpOutBuffer,
__in DWORD nOutBufferSize,
__out_opt LPDWORD lpBytesReturned,
Using these, you can give data to the driver (through the lpInBuffer pointer and nInBufferSize parameter specifying how big it is - what this data is or how to interpret it is up to your driver) and the kernel-mode layer can give data back through lpOutBuffer (a pointer to memory you, in user mode, have already allocated - this is not a pointer that the driver sets!), its size in bytes in nOutBufferSize (again you know this, since you in user mode allocate this buffer) and then the driver will tell you how much of that buffer it actually filled with lpBytesReturned.
This Wikipedia article describes the general concept of ioctl functions, of which DeviceIOControl is an example.
Note: You said "I am trying to share memory between user space and kernel space in windows". This is not exactly shared memory - it's not memory that both user mode and kernel mode are reading from or writing to at the same time, for example. It is memory where, for the duration of the DeviceIOControl function call, kernel mode has access to your user mode-allocated memory for which you pass it pointers (although from memory it's slightly more complicated than that, but that's the effect.) Ie it's only 'shared' while you call that method.
2) Another alternative is to use ReadFile and WriteFile if you only need simple data transfer and the kernel driver accepts it. It doesn't allow you to have two-way communication quite as DeviceIOControl does (where you give the driver data and it gives you something back with an error code) but it's simple and you're probably already familiar with those APIs.
Kernel mode side: Are you writing the kernel driver as well? If so, this article has information about implementing the kernel-mode side of IOCTL. This article series also describes allowing user mode to use ReadFile and WriteFile to communicate if you choose that methods.
A process ID is a number that uniquely identifies a process. A process handle is also a number that uniquely identifys a process kernal object.
Why do we need them both since either of them can identify a process.
I think the answer may lie in the mapping relationship betweeen a process and a process kernel object. Is it true that more than one process kernel objects can be mapped to a single process? And each process kernel object has its own process handle. So that each of the process kernel object could represent different access mode or things like that.
This question came to me when I am using the MiniDumpWriteDump() function, which is declared like this:
BOOL WINAPI MiniDumpWriteDump(
__in HANDLE hProcess,
__in DWORD ProcessId,
__in HANDLE hFile,
__in MINIDUMP_TYPE DumpType,
__in PMINIDUMP_EXCEPTION_INFORMATION ExceptionParam,
__in PMINIDUMP_USER_STREAM_INFORMATION UserStreamParam,
__in PMINIDUMP_CALLBACK_INFORMATION CallbackParam
);
So it's parameters include both process id and process handle. I just don't know why it's necessary to have them both.
Many thanks for your insights.
Process Handle is
Arbitrary
Internal to process that acquired it. Private and can't be shared between threads/processes
It carries security access rights too
While Process ID is
Unique
Universal, public, so it can be shared between threads/processes
the difference is that 'id' is system-wide number which uniquely identifies the process. 'handle' on the other hand is an opaque value which connects process and access to that process to your program. you can potentially have multiple different handles to the same process.
I don't know why MiniDumpWriteDump takes both.
OK, so I have a very large multi-threaded unmanaged c++ application (server) that runs on a windows 2003 server. It hosts sessions for 20-50 concurrent users doing all sorts of business logic... At times it can be using a very large amount of memory due to things like object/session caching due to users having large numbers of windows open in the clients (each window has a separate server 'session'.
We routinely see consumption of more than 5-600 MB physical memory and 5-600 MB of virtual memory. Once it gets to this point we seem to start having 'out of memory' errors.
Now I know that Windows will start page-faulting when it feels it needs to free up physical memory, and also that win32 applications normally are only able to allocate up to a maximum of 4GB worth of memory, really only with 2GB of that available for actual use by the application for 'user-mode' address space, and even less of that after other libraries are loaded... I'm not sure if the 'user-mode' memory usage is what is reported on the Task Manager...
So anyway my real question is:
How can I find out how much user-mode memory my application has access to, and how much has been used at any given time (preferably from outside of the application, i.e. some windows management tool)?
[edit] After looking at the Process Explorer and the website, it looks like the value 'Virtual Size' is the value of how much memory the application has access to.
Sounds like a case for Process Explorer, a free utility from Microsoft SysInternals:
(source: microsoft.com)
Description:
Ever wondered which program has a
particular file or directory open? Now
you can find out. Process Explorer
shows you information about which
handles and DLLs processes have opened
or loaded.
The Process Explorer display consists
of two sub-windows. The top window
always shows a list of the currently
active processes, including the names
of their owning accounts, whereas the
information displayed in the bottom
window depends on the mode that
Process Explorer is in: if it is in
handle mode you'll see the handles
that the process selected in the top
window has opened; if Process Explorer
is in DLL mode you'll see the DLLs and
memory-mapped files that the process
has loaded. Process Explorer also has
a powerful search capability that will
quickly show you which processes have
particular handles opened or DLLs
loaded.
The unique capabilities of Process
Explorer make it useful for tracking
down DLL-version problems or handle
leaks, and provide insight into the
way Windows and applications work.
If you are looking for more info in terms of terminal-server specific info, I've been following the blog of a programmer that is releasing a beta of a tool that I believe will fit your needs perfectly. It is an improved TSAdmin. He calls it TSAdminEx.
See below for a screenshot, and click here to learn more about it and to get the beta. It's free software, BTW.
I know you asked for preferably from outside of the application, but I was googling for how to find such information from within my own program and stumbled upon your post. So, this is to benefit people who want this information from within their program.
unmanaged C++
#include <windows.h>
#include <stdio.h>
#include <psapi.h>
void PrintMemoryInfo( DWORD processID )
{
HANDLE hProcess;
PROCESS_MEMORY_COUNTERS pmc;
// Print the process identifier.
printf( "\nProcess ID: %u\n", processID );
// Print information about the memory usage of the process.
hProcess = OpenProcess( PROCESS_QUERY_INFORMATION |
PROCESS_VM_READ,
FALSE,
processID );
if (NULL == hProcess)
return;
if ( GetProcessMemoryInfo( hProcess, &pmc, sizeof(pmc)) )
{
printf( "\tPageFaultCount: 0x%08X\n", pmc.PageFaultCount );
printf( "\tYour app's PEAK MEMORY CONSUMPTION: 0x%08X\n",
pmc.PeakWorkingSetSize );
printf( "\tYour app's CURRENT MEMORY CONSUMPTION: 0x%08X\n", pmc.WorkingSetSize );
printf( "\tQuotaPeakPagedPoolUsage: 0x%08X\n",
pmc.QuotaPeakPagedPoolUsage );
printf( "\tQuotaPagedPoolUsage: 0x%08X\n",
pmc.QuotaPagedPoolUsage );
printf( "\tQuotaPeakNonPagedPoolUsage: 0x%08X\n",
pmc.QuotaPeakNonPagedPoolUsage );
printf( "\tQuotaNonPagedPoolUsage: 0x%08X\n",
pmc.QuotaNonPagedPoolUsage );
printf( "\tPagefileUsage: 0x%08X\n", pmc.PagefileUsage );
printf( "\tPeakPagefileUsage: 0x%08X\n",
pmc.PeakPagefileUsage );
}
CloseHandle( hProcess );
}
int main( )
{
PrintMemoryInfo( GetCurrentProcessId() );
return 0;
}
You wrote:
When your talking how much memory a
win32 app can access they specifically
call it 'user-mode' memory which I
don't see as an option or at least I
don't know what column it really is.
Have a look at this article (written by the creator of Process Explorer, Dr. Mark Russinovich).
To be able to manage your Windows systems effectively you need to understand how Windows manages physical resources, such as CPUs and memory, as well as logical resources, such as virtual memory, handles, and window manager objects. Knowing the limits of those resources and how to track their usage enables you to attribute resource usage to the applications that consume them, effectively size a system for a particular workload, and identify applications that leak resources.
What is the win32 API function for private bytes (the ones you can see in perfmon).
I'd like to avoid the .NET API
BOOL WINAPI GetProcessMemoryInfo(
__in HANDLE Process,
__out PPROCESS_MEMORY_COUNTERS ppsmemCounters,
__in DWORD cb
);
Where ppsmemCounters parameter can be a PROCESS_MEMORY_COUNTERS or PROCESS_MEMORY_COUNTERS_EX structure. Just typecast PROCESS_MEMORY_COUNTERS_EX to PROCESS_MEMORY_COUNTERS.
PROCESS_MEMORY_COUNTERS_EX.PrivateUsage is what you're looking for.
More info here and here
You can collect the same data perfmon shows using the performance counters API
You need to clarify what you are trying to do. These are internal figures whose value is not really controlled by any API.
Technically Private Bytes is the commit charge, the amount of memory allocated in the swap file to hold the contents of the applications private memory should it be swapped out.
Generally private bytes = amount of dynamically allocated memory + some extra.