Is there a size limit to the file mapping object? The reason I'm asking is that there is a mentioning of 2GB limit somewhere in MSDN (lost the track..) and I also checked this sample, which also expects 2GB size limit:
https://cpp.hotexamples.com/examples/-/-/CreateFileMapping/cpp-createfilemapping-function-examples.html
But I tried on a 40GB file with no problems on newest Win 10, so I'm a bit worried if there wasn't some limitation on older Windows for example.
There is no 2GB limit for file mappings. You can boot 32-bit Windows with the 3GB option or when a 32-bit process running on a 64-bit system, you get the full 4GB if the correct PE flag is set. All these limits are theoretical and you will never reach them in practice.
How large of a view you can map depends on two things;
The contiguous range of free addresses in your processes address space.
Available kernel memory for keeping track of the memory pages.
The first one is the big limit on 32-bit systems since the address space of your process is shared with system libraries, 3rd-party libraries (anti-virus, injected "tweaking" tools etc.), the PEB and TEBs, the system region, thread stacks and memory reserved by hardware. This will often put you well below 2GB. Any design requiring more than 500MB should probably be changed to only map in specific smaller ranges as needed.
For a 64-bit process on 64-bit Windows, the virtual address space is the 128-terabyte range 0x000'00000000 through 0x7FFF'FFFFFFFF (KB889654 claims 8 TB but that only applies to < Windows 8.1). Any usable range is going to be smaller but you can assume a couple of terabyte at least. 40GB is no problem and not enough to run into problems with low system resources either.
Related
Hi people I was wondering about something called addressing schemes that the operating systems use for expandable RAM's. Let us consider an example to throw some light on this.
"If we have a 32 bit architecture for the computer then this means we have a computer address that is 32-bit long which amounts to 2^32 addressable memory location approximately 4GB of data."
But if we add another 4GB of main memory that is now 8GB of RAM in effect, how does the computer address the extra main memory locations because that additional amount exceeds the range of a 32 bit address which is 2^32.
Can anyone throw some light on this question.
Basically, you can't address 8GB with just 32bits. At any given point in time using 32bits you can only choose from 4G memory locations .
A popular workaround is to use physical addresses which are larger than 32 bits in the page tables. This allows the operating system to define which subset of the 8GB a program is able to access. However, this subset can never be bigger than 4GB. x86 PAE is one example, but there are others which do just the same.
With this workaround the operating system itself can access the whole 8GB only by changing it's own page table. E.g. to access a memory location, it first has to map the memory location into its own address space by changing the page table, and only then can start accessing the memory location. Of course this is very cumbersome (to say the least). It can also lead to problems if parts of the operating system were written without considering this type of memory extension, device drivers are a typical example.
The problem is not new. 8bit Computers like Commodores C64 used bank switching to access more than 64KB with 16bit addresses. Early PCs used expanded memory to work around the 640KB limit. The Right Thing (TM) is of course to switch to bigger addresses before you have to resort to ugly solutions.
What is the maximum amount of memory any single process on Windows can address? Is this different than the maximum virtual memory for the system? How would this affect a system design?
On 32-bit versions of Windows, a single process can map and address no more than 3GB of virtual memory at time. In 64-bit versions of Windows, a 32-bit process can map and address no more than 4GB of virtual memory at a time.
For 64-bit processes, the amount is difficult to calculate as there are numerous overlapping limits that could apply depending on all kinds of factors. It's typically around 7TB.
The maximum amount of virtual memory for the system is difficult to calculate and not a very meaningful number. Also, the limits on physical memory are not related to these limits on virtual memory.
You can read more details on Microsoft's page, Memory Limits for Windows Releases.
When no LARGEADDRESSAWARE switch is given in a 32bit executable, 2GB of memory (give or take) is available the the process to use. When the switch LARGEADDRESSAWARE is present in the PE flags of the executable this limit can be (correct me if I am wrong):
2GB if a 32 bit Windows was not started with the /3GB switch
3GB if a 32 bit Windows was started with the /3GB switch
almost up to 4GB if the process runs under a Windows 64 bit OS as a 32 bit process.
My question is: how can one determine this memory limit (with and/or without the LARGEADDRESSAWARE flag)? And as a sidenote: is the enumeration of possibilities above correct?
Note: I am not interested in the amount of memory the process is using, also not the limit due to external effects, just the maximum amount of memory I can allocate in the ideal case.
I think the best approach is to call GetSystemInfo and work out what you need from lpMinimumApplicationAddress and lpMaximumApplicationAddress. You can simply subtract the former from the latter to obtain the total available addressable memory space.
Your three bullet points of the various possibilities are correct.
I've been reading that 32bit Windows applications are limited to 2 GB RAM because the upper 2GB of addressing space is reserved for the Windows OS (and, iirc, VRAM). If you use the /3GB flag on 32-bit WinXp you might get up to 3 GB of RAM available for addressing, but usually you have to tweak with userva values. I've heard that on 64 bit editions of Windows, with a large address aware flag in the PE header and over 4 GB of RAM, it is possible for an application to use all 4 GB of addressing space for its own memory management.
On the other hand, I'm pretty sure that when you call the windows API, you have to call memory locations within the 32-bit address space you're provided. So, exactly how much RAM can a 32-bit large address aware application use for itself in a 64-bit environment, really? And why?
Thank you.
The virtual address space is extended to 4GB. If you don't use the Address Windowing Extension API, the maximum amount of memory you can access is 4GB. Some of that space will be taken up by the OS for .dlls and other such things, but it will be possible for you to get memory back that uses all 32-bits of a pointer.
Incidentally, if you aren't large address aware, all memory pointers will not be negative when cast to a INT_PTR. This is actually a source of more than a few subtle bugs when using the large address aware flag, as pointers are treated signed values.
I am playing with an MSDN sample to do memory stress testing (see: http://msdn.microsoft.com/en-us/magazine/cc163613.aspx) and an extension of that tool that specifically eats physical memory (see http://www.donationcoder.com/Forums/bb/index.php?topic=14895.0;prev_next=next). I am obviously confused though on the differences between Virtual and Physical Memory. I thought each process has 2 GB of virtual memory (although I also read 1.5 GB because of "overhead". My understanding was that some/all/none of this virtual memory could be physical memory, and the amount of physical memory used by a process could change over time (memory could be swapped out to disc, etc.)I further thought that, in general, when you allocate memory, the operating system could use physical memory or virtual memory. From this, I conclude that dwAvailVirtual should always be equal to or greater than dwAvailPhys in the call GlobalMemoryStatus. However, I often (always?) see the opposite. What am I missing.
I apologize in advance if my question is not well formed. I'm still trying to get my head around the whole memory management system in Windows. Tutorials/Explanations/Book recs are most welcome!
Andrew
That was only true in the olden days, back when RAM was expensive. The operating system maps pages of virtual memory to RAM as needed. If there isn't enough RAM to satisfy a program's request, it starts unmapping pages to make room. If such a page contains data instead of code, it gets written to the paging file. Whenever the program accesses that page again, it generates a paging fault, letting the operating system read the page back from disk.
If the machine has little RAM and lots of processes consuming virtual memory pages, that can cause a very unpleasant effect called "thrashing". The operating system is constantly accessing the disk and machine performance slows down to a crawl.
More RAM means less disk access. There's very little reason not to use 3 or 4 GB of RAM on a 32-bit operating system, it's cheap. Even if you do not get to use all 4 GB, not all of it will be addressable due hardware devices taking space on the address bus (video, mostly). But that won't change the size of the virtual memory accessible by user code, it is still 2 Gigabytes.
Windows Internals is a good book.
The amount of virtual memory is limited by size of the address space - which is 4GB per process on a 32-bit system. And you have to subtract from this the size of regions reserved for system use and the amount of VM used already by your process (including all the libraries mapped to its address space).
On the other hand, the total amount of physical memory may be higher than the amount of virtual memory space the system has left free for your process to use (and these days it often is).
This means that if you have more than ~2GB or RAM, you can't use all your physical memory in one process (since there's not enough virtual memory space to map it to), but it can be used by many processes. Note that this limitation is removed in a 64-bit system.
I don't know if this is your issue, but the MSDN page for the GlobalMemoryStatus function contains the following warning:
On computers with more than 4 GB of memory, the GlobalMemoryStatus function can return incorrect information, reporting a value of –1 to indicate an overflow. For this reason, applications should use the GlobalMemoryStatusEx function instead.
Additionally, that page says:
On Intel x86 computers with more than 2 GB and less than 4 GB of memory, the GlobalMemoryStatus function will always return 2 GB in the dwTotalPhys member of the MEMORYSTATUS structure. Similarly, if the total available memory is between 2 and 4 GB, the dwAvailPhys member of the MEMORYSTATUS structure will be rounded down to 2 GB. If the executable is linked using the /LARGEADDRESSAWARE linker option, then the GlobalMemoryStatus function will return the correct amount of physical memory in both members.
Since you're referring to members like dwAvailPhys instead of ullAvailPhys, it sounds like you're using a MEMORYSTATUS structure instead of a MEMORYSTATUSEX structure. I don't know the consequences of that on a 64-bit platform, but on a 32-bit platform that definitely could cause incorrect memory sizes to be reported.