We are working on a Vista/Windows 7 application that will be running in 64 bit mode using VS2008/C++. We will be needing to cache hundreds of 2-3 mb blobs of data in RAM for performance reasons up to some memory limit. Our usage profile is such that we cannot read the data in fast enough if it is all on the the disk. Cached Memory usage will be larger than 1gb memory used. For this to work well, we need to ensure that Windows does not page this memory out as it will defeat the purpose of why we are doing this.
I've done a fair amount of research and cannot find documentation that states exactly how to do this. I've seen several references that infer memory mapped files work this way. Is there an expert who can clarify this for me?
I'm aware there are other programs that we could adapt to do this, for example, splitting the blobs and loading into memcache or inmemory databases, but they all have too many problems with performance or code complexity.
Suggestions?
You can use VirtualLock. However, you'll surely hit the quota with the amount you're talking about. Given that you should never run any other code on this machine, you'll be better off just disabling the paging file. Control Panel + System + Advanced.
From user mode, you can't (EDIT: At least for the sizes you're talking about). User mode allocations all come down to either the VirtualAlloc API (On top of which the GlobalAlloc/LocalAlloc/C Runtime's functions are written) or the Memory Mapped File API. Neither API supports this, and therefore it's impossible to obtain on Win32. It is possible from whithin Kernel Mode, but somehow I suspect this is a user-mode application :)
Note that the memory manager is not going to decide to page your RAM without good reason to do so.
Now, you could of course, if you control the machine completely (this is for internal use or something) disable the pagefile on the machine in question, but that does not seem to solve your problem.
It's possible! You can force pages to be locked in memory from a user mode app by allocating them using AWE (Address Windowing Extensions) VirtualAlloc + AllocatePhysicalPages + MapPhysicalPages.
Note: I have read that you can use the AWE APIs from either a 32-bit or 64-bit app also, but I've only tried with 32-bit app. (Of course since it's AWE you can manually remap memory to access > 2GB RAM.)
Note: You have to first have seLockMemoryPrivilege. (Which seems to require the app to run as Administrator in my testing so far.)
Note: Using AWE implies some limitations on what you can do with those particular pages of memory, e.g. no VirtualProtect().
perhaps the answer? (from a VMWARE tutorial)
To edit the Registry and disable paging kernel-mode stacks
Click Start > Run and type regedit.
In the left pane of the Registry Editor, navigate to
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager.
In the right pane, rightâclick GlobalFlag and select Modify.
With Base Hexadecimal, type value 80000, which corresponds to FLG_DISABLE_PAGE_KERNEL_STACKS.
Click OK and exit the Registry Editor.
Reboot the guest system for this change to take effect.
hope it helps
Related
Back in the good old/bad old days when I developed on VAX/VMS it had a feature called 'Installed Shared Images' whereby if one expected one's executable program would be run by many users concurrently one could invoke the INSTALL utility thus:
$ INSTALL
INSTALL> ADD ONES_PROGRAM.EXE/SHARE
INSTALL> EXIT
The /SHARE flag had the effect of separating out the code from the data so that concurrent users of ONES_PROGRAM.EXE would all share the code (on a read-only basis of course) but each would have their own copy of the data (on a read-write basis). This technique/feature saved Mbytes of memory (which was necessary in those days) as only ONE copy of the program's code ever needed to be resident in VAX memory irrespective of the number of concurrent users.
Does Windows XP have something similar? I can't figure out if the Control Panel's 'Add Programs/Features' is the equivalent (I think it is, but I'm not sure)
Many thanks for any info
Richard
p.s. INSTALL would also share Libraries as well as Programs in case you were curious
The Windows virtual memory manager will do this automatically for you. So long as the module can be loaded at the same address in each process, the physical memory for the code will be shared between each process that loads that module. That is true for all modules, libraries as well as executables.
This is achieved by the linker marking code segments as being shareable. So, linkers mark code segments as being shareable, and data segments otherwise.
The bottom line is that you do not have to do anything explicit to make this happen.
Is it possible in Win7 family to read a random known location at lower 4gb of RAM from userspace application? The memory at that location is allocated by kernel driver.
I would seriously hope that this is not possible, neither under Windows 7 nor any (reasonably recent, such as NT/2k) previous version.
If this was possible, any unprivilegued random snippet of executable code that gets on your computer somehow could trivially compromise system security, for example by reading out the drive encryption key, or authentication data of users as they log in, and whatnot.
It would not make any sense to run a webbrowser under a limited user account. Any exploit could just read sensible information from memory anyway. What programs are running, which users are logged in (and other info such as credential and encryption keys), what the firewall settings are etc etc.
Of course, as Sergius said, you can always install a driver and remap memory regions so they are accessible, but there's not much the OS can do against that if you've allowed the installation of a driver. That's why that annoying little "do you really want?" box pops up.
As I know it is impossible without a driver help in Win7. From a driver you can map a required memory to a user-mode address space or provide IOCLT for memory reading.
Windows Server 2008. How can I quickly use up RAM so to induce GC in my app. If there is a way to do it without needing Visual Studio or installing a language runtime it would be good.
EDIT: I don't want to have to write an app and then copy it over to the server. I'm looking for a way to do it quickly without writing an app that requires an IDE or installation of a runtime/compiler.
Perhaps a powershell or batch script?...
I don't think using up RAM outside your process is going to necessarily trigger GC.
If I understand your question correctly, you have a program Foo.exe that is written in some unknown language, running on some unknown runtime (are you not allowed to post the details for some reason, or do you just not know?), and you want to try to get that program's runtime to trigger a garbage collection. However, you want to do this by using up RAM outside of foo.exe.
You could do this by creating a simple batch file that just started up a hundred copies of IE or Word or whatever program you want. However, I don't think that will do what you want it to do. If your process has already allocated a certain amount of memory, it won't necessarily give that memory up or trigger GC just because other processes are being started. It may page to disk, or may force other programs to page to disk. But not all Garbage Collectors are alike, so we can't really help without more details. I'm pretty sure some VM's never give back memory once they've allocated it, even after GC.
You could run your program inside a virtual machine such as Virtual Box, where you specify the memory ceiling of the guest operating system.
I'm having trouble imagining a scenario where this would be necessary though. Could you provide more information about the problem?
If you are using java you can specify the max amount of memory using Xmx. Search for JVM memory setting
We have some Win32 console applications running on Windows Server 2003 Service Pack 2 that regularly fail with this:
Error 1450 (ERROR_NO_SYSTEM_RESOURCES): "Insufficient system resources exist to complete the requested service."
All the documentation we've found suggests it is linked to the number of Free System Page Table Entries running out. We have 16GB RAM in these machines and use the /3GB Operating System switch to squeeze the Windows kernel into 1GB and allow our processes access to 3GB of address space. This drastically reduces the total number of Free System Page Table Entries, so combined with our heavy use of MapViewOfFile() it is perhaps not surprising that the kernel page table entries are running out.
However, when using Performance Monitor to view the Free System Page Table Entries counter, the value is around 36,000 on reboot and doesn't go down when our application starts. I find it hard to believe that our application, which opens many large memory-mapped files, doesn't have any effect on the kernel page table. If we can't believe the counter, it's much more difficult to test the effect of any system changes we make.
There is a promising Knowledge Base article, The Performance tool does not accurately show the available Free System Page Table entries in Windows Server 2003, but it says the problem has been fixed in Service Pack 1, and we are already on Service Pack 2.
Has anyone else struggled with or solved this issue?
Update: I have checked !sysptes in windbg (debugging the kernel) and the value matches the performance counter, around 36,000. I guess this is most likely to mean that there really are that many free page table entries and Windows is telling the truth. It does leave the question of why we're getting 1450 errors though, if the PTEs are not running out.
Further update: We never did get to the bottom of why the 1450 errors were occurring. However, instead we upgraded the OS on these servers to 64-bit Windows. This allows the existing 32-bit applications (without recompilation) to access a full 4GB of virtual address space, and lets the kernel memory area with those pesky Page Table Entries be as big as it likes too. I don't think we've had a 1450 error since.
Can you try the windbg command "!sysptes" to get System PTE Information? I'm not sure if you can do this with live kernel debug, you may have to get a memory dump.
I'm not sure why you assume that ERROR_NO_SYSTEM_RESOURCES is caused only by running out of free System Page Table Entries ? As far as I know, such generic error codes are used for more than one resource type. And in fact, the first Google hit suggests that running out of file cache memory may cause it too. (KB on an XP bug, which tripped this error mode).
In your case, I'd be checking the "Handle Count". Another possible problem is address space fragmentation. If you you want to create a 1GB file mapping view, you need 1GB of free address space, and it has to be contiguous. If you map a 1GB file, a 800 MB file, and a 1GB file, close the 800MB one and open a 900MB file, the 900MB file may not fit in the hole that's left.
MS has 2 ways to allow there 32 bit OS to "deal" with hardware that has 4 GB or more of RAM.
Option 1: is what you did with the /3GB Switch in the Boot.ini.
Option 1 Pros and Cons:
(CONS) This option sucks 1 GB from the normal 2 GB kernel area - hence making the OS struggle to meet the demands of both Paged Pool allocations and kernel stack allocations. So a person might think that using the /3GB Switch will help their, but really this option is screwing the 32 bit Window OS into a slow death.
(CONS) But, This gives my App 3GB.... WRONG (Hence this is a CON) The catch is that ONLY application that have been recompiled from the vendor to be "/3GB Switch aware" can really use the extra 1 GB. Hence the whole use of the /3GB Switch is a really BAD J.O.K.E on everyone.
Read this link for a much better write-up:
http://blogs.technet.com/askperf/archive/2007/03/23/memory-management-demystifying-3gb.aspx
Option 2: Use the /PAE switch in the Boot.ini.
Option 2 Pros and Cons:
(PROS) This really this only option if you have a more then 4GB of RAM. It tricks a application by placing the complete application memory footprint in RAM. Normally, only a application "Working Set" memory is in RAM and the remaining application memory requirements go into Windows Pagefile. What is a application total memory requirements?? - it called "Virtual Size".
In my world, I have a big fat Java based IBM Product that I deal with. The server that is running the "application" has 16 GB of RAM. I simply add the /PAE switch and watch (thanks to sysinternals Processes Explorer) application paging requests go from 200 KB per sec to up to 4MB per sec.
Question: "Why"?
Answer: The whole application is in RAM.
Question: "Does the application know that it is completely running in RAM?
Answer: No - It is running that same old way that it was always run, "THINKING" that it's has part of itself as the "Working Set" memory living in RAM and the remaining application memory requirements go into Windows Pagefile.
Yes, it is that flipping GOOD.
Please Note: Microsoft has done a poor job telling anyone about the great Windows OS option. Duh
Try it and report back to stackoverflow....
I need to modify the MBR of Windows, and I would really like to do this from Windows.
Here are my questions. I know that I can get a handle on a physical device with a call to CreateFile. Will the MBR always be on \\.\PHYSICALDRIVE0? Also, I'm still learning the Windows API to read directly from the disk. Is readabsolutesectors and writeabsolutesectdors the two functions I'm going to need to use to read/write to the disk sectors which contain the MBR?
Edit from from what I've learned on my own.
The MBR will not always be on \\.\PHYSICALDRIVE0. Also, you can write to the bootsector (at least as Administrator on XP) by call CreateFile with the device name of the drive that contains the MBR. Also, you can write to this drive by simply calling WriteFile and passing the handle of the device created by calling CreateFile.
Edit to address Joel Coehoorn.
I need to edit the MBR because I'm working on a project that needs to modify hardware registers after POST in BIOS, but before Windows will be allowed to boot. Our plan is to make these changes by modifying the bootloader to execute our code before Windows boots up.
Edit for Cd-MaN.
Thanks for the info. There isn't anything in your answer, though, that I didn't know and your answer doesn't address my question. The registry in particular absolutely will not do what we need for multiple reasons. The big reason being that Windows is the highest layer among multiple software layers that will be running with our product. These changes need to occur even before the lower levels run, and so the registry won't work.
P.S. for Cd-MaN.
As I understand it, the information you give isn't quite correct. For Vista, I think you can write to a volume if the sectors being written to are boot sectors. See http://support.microsoft.com/kb/942448
Once the OS is started the MBR is typically protected for virus reasons - this is one of the oldest virus tricks in the books - goes back to passing viruses from floppy to floppy.
Even if it wasn't restricted, you have to write low level code - it isn't part of the file system, but exists on a specific location on the hard drive.
Due to that, you pretty much are restricted to writing low level (most programs implement this in assembly) or C code targeting 16 bit DOS.
Most of these programs use the BIOS interface (13h, I believe) to access the sectors of the disk directly. You can access these in C using some inline assembly, or compiler provided interfaces. You will generally not get access to BIOS without the cooperation of the OS, though, so your program, again, will be restricted to DOS. If you can access these you're almost home free - the nice thing about BIOS is you don't have to worry about what type of HD is in the system - even RAID cards often insert themselves into the BIOS routines so they can be accessed without knowing where in memory the ATA or SATA controller is, and executing commands on that low level.
If you absolutely must access it within an OS, though, you pretty much have to write a device driver to access the BIOS or the memory space where the HD controllers exist. I wouldn't recommend it, though, as this is very tricky to deal with - modern computers put the HD controllers in different spots in memory, with different IRQs, and each chipset has become a little more esoteric because they can provide a minimum interface to bios for bootup, and then a specific driver for Windows. They skip all the other interface niceties that would be considered compatible with other controllers because it's more expensive to be compatible.
You may find that at the driver level inside windows you'll have methods for accessing the drive sectors directly (or pseudo directly), but again, they are likely very well protected due to the aforementioned virus issues.
Good luck!
Modifying the bootloader is bad, bad idea. Here are just a few of the possible gotcha's:
it will potentially kill full disk encryption products (Truecrypt, PGP, Vista's BitLocker, etc)
it will potentially trip up AV products (scaring users)
it will potentially kill complicated booting scenarios (chained boot loaders, etc)
it will kill off the chain of trust when using the TPM module (because it checks the MBR for change before executing it)
direct disk access is not allowed starting from Vista (only using drivers)
Alternatives (like modifying the hardware register during the Windows bootup via a driver which is set to load at boot time or after Windows has booted) should really be considered. If the modification is as simple as writing to a port, ie:
OUT AX, BL
then drivers exists for all versions of Window which can do this (reading/writing a value from/to a certain port) which can be called from user mode.
Maybe a PXE boot scenario could help you? Simply boot on your crafted PXE image which modify the hardware registers you need to modify, and then return the control to the Master Boot Record or to the active partition's boot record.
This way you don't have to modify the boot records.