MapViewOfFile() no longer works after process hits the 2GB limit - winapi

Question

MapViewOfFile() works without any problem if our process has not hit the 2GB limit yet. However if the process hits the limit then MapViewOfFile() no longer works even if some or all of the memory is deallocated. GetLastError() returns 8, which means ERROR_NOT_ENOUGH_MEMORY, Not enough storage is available to process this command. Here is a small program showing the problem:
#include <Windows.h>
#include <cstdio>
#include <vector>
const int sizeOfTheFileMappingObject = 20;
const int numberOfBytesToMap = sizeOfTheFileMappingObject;
const char* fileMappingObjectName = "Global\\QWERTY";
void Allocate2GBMemoryWithMalloc(std::vector<void*>* addresses)
{
const size_t sizeOfMemAllocatedAtOnce = 32 * 1024;
for (;;) {
void* address = malloc(sizeOfMemAllocatedAtOnce);
if (address != NULL) {
addresses->push_back(address);
}
else {
printf("The %dth malloc() returned NULL. Allocated memory: %d MB\n",
addresses->size() + 1,
(addresses->size() * sizeOfMemAllocatedAtOnce) / (1024 * 1024));
break;
}
}
}
void DeallocateMemoryWithFree(std::vector<void*>* addresses)
{
std::vector<void*>::iterator current = addresses->begin();
std::vector<void*>::iterator end = addresses->end();
for (; current != end; ++current) {
free(*current);
}
addresses->clear();
printf("Memory is deallocated.\n");
}
void TryToMapViewOfFile()
{
HANDLE fileMapping = OpenFileMappingA(FILE_MAP_ALL_ACCESS, FALSE,
fileMappingObjectName);
if (fileMapping == NULL) {
printf("OpenFileMapping() failed. LastError: %d\n", GetLastError());
return;
}
LPVOID mappedView = MapViewOfFile(fileMapping, FILE_MAP_READ, 0, 0,
numberOfBytesToMap);
if (mappedView == NULL) {
printf("MapViewOfFile() failed. LastError: %d\n", GetLastError());
if (!CloseHandle(fileMapping)) {
printf("CloseHandle() failed. LastError: %d\n", GetLastError());
}
return;
}
if (!UnmapViewOfFile(mappedView)) {
printf("UnmapViewOfFile() failed. LastError: %d\n", GetLastError());
if (!CloseHandle(fileMapping)) {
printf("CloseHandle() failed. LastError: %d\n", GetLastError());
}
return;
}
if (!CloseHandle(fileMapping)) {
printf("CloseHandle() failed. LastError: %d\n", GetLastError());
return;
}
printf("MapViewOfFile() succeeded.\n");
}
int main(int argc, char* argv[])
{
HANDLE fileMapping = CreateFileMappingA(INVALID_HANDLE_VALUE, NULL,
PAGE_READWRITE, 0, sizeOfTheFileMappingObject, fileMappingObjectName);
if (fileMapping == NULL) {
printf("CreateFileMapping() failed. LastError: %d\n", GetLastError());
return -1;
}
TryToMapViewOfFile();
std::vector<void*> addresses;
Allocate2GBMemoryWithMalloc(&addresses);
TryToMapViewOfFile();
DeallocateMemoryWithFree(&addresses);
TryToMapViewOfFile();
Allocate2GBMemoryWithMalloc(&addresses);
DeallocateMemoryWithFree(&addresses);
if (!CloseHandle(fileMapping)) {
printf("CloseHandle() failed. LastError: %d\n", GetLastError());
}
return 0;
}
The output of the program:
MapViewOfFile() succeeded.
The 65126th malloc() returned NULL. Allocated memory: 2035 MB
MapViewOfFile() failed. LastError: 8
Memory is deallocated.
MapViewOfFile() failed. LastError: 8
The 64783th malloc() returned NULL. Allocated memory: 2024 MB
Memory is deallocated.
As you can see MapViewOfFile() fails with 8 even after releasing all memory that was allocated. Even though MapViewOfFile() reports ERROR_NOT_ENOUGH_MEMORY we can call malloc() successfully.
We ran this example program on Windows7,32bit; Windows 8.1,32bit and Windows Server 2008 R2,64bit and the results were the same.
So the question is: Why does MapViewOfFile() fail with ERROR_NOT_ENOUGH_MEMORY after the process hits the 2GB limit?

Answer 1

Why MapViewOfFile fails
As IInspectable's comment explains freeing memory allocated with malloc doesn't make it available for use with MapViewOfFile. A 32-bit processes under Windows has a 4 GB virtual address space and only the first 2 GB of it is available for the application. (An exception would be a large address aware program, which increases this to 3 GB under suitably configured 32-bit kernels and to 4 GB under 64-bit kernels.) Everything in your program's memory must have a location somewhere in this first 2 GB of virtual address space. That includes the executable itself, any DLLs used by the program, any data you have in memory, whether allocated statically, on the stack or dynamically (eg. with malloc), and of course any file you map into memory with MapViewOfFile.
When your program first starts out the Visual C/C++ runtime creates a small heap for dynamic allocations for functions like malloc and operator new. As necessary the runtime increases the size of the heap in memory, and as it does so it uses up more virtual address space. Unfortunately it never shrinks the size of the heap. When you free a large block of memory the runtime only decommits the memory used. This makes the RAM used by the freed memory available for use, but virtual address space taken up by the freed memory remains allocated as part of the heap.
As mentioned previously a file mapped into memory by MapViewOfFile also takes up virtual address space. If the heap (plus your program, DLLs, and everything else) are using up the all virtual address space then there's no room to map files.
A possible solution: don't use malloc
An easy way to avoid the heap from growing to fill all of the virtual address space is to not to use the Visual C/C++ runtime to allocate large (at least 64k) blocks of memory. Instead allocate and free memory from Windows directly using VirtualAlloc(..., MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE) and VirtualFree(..., MEM_RELEASE). The later function releases both the RAM used by the region and virtual address space taken up by it, making it available for use with MapViewOfFile.
But...
You can still run into another problem, where MapViewOfFile can still fail even though the size of view is smaller, sometimes even much smaller, than the total amount free virtual address space. This is because the view needs to be mapped into a contiguous region of virtual address space. If the virtual address space becomes fragmented. the largest contiguous region of unreserved virtual address space can up being relatively small. Even when your program first starts up, before you have had a chance to do any dynamic allocations, the virtual address space can be somewhat fragmented because of DLLs loaded at various addresses. If you have a long lived program that does a lot of allocations and deallocations with VirtualAlloc and VirtualFree, you can end up with a very fragmented virtual address space. If you encounter this problem you'll have to change your pattern of allocations, maybe even implement your own heap allocator.