I want to share memory between processes using mmap(). mmap() can be applied on a regular file, or on a POSIX shared memory file (shm_open). But what is the difference? The difference is surely not whether the disk is accessed:
Frequently and recently accessed regular files live in the filesystem cache/buffer so their contents are already in memory.
Even when using mmap() with shm_open(), that memory region may be swapped out when the system is under high memory stress.
One difference is that although shared memory might be written to disk, it will never be written to disk permanently. If you create a normal file, then it persists across reboots. If you create a shared memory object, it doesn't. It behaves quite like a temporary file in that respect.
Another difference is that "It is unspecified whether the name appears in the file system and is visible to other functions that take pathnames as arguments". So, although a shared memory object has a file-like path, it need not actually appear in the filesystem.
Really, both of those differences are about decoupling from the filesystem. Files and shared memory objects are both bags of bytes which can be read and written, but files differ in that they are persisted in the filesystem.
Related
outI'm working with memory mapped files (MMF) with very large datasets (depending on the input file), where each file has ~50GB and there are around 40 files open at the same time. Of course this depends, I can also have smaller files, but I can also have larger files - so the system should scale itself.
The MMF is acting as a backing buffer, so as long as I have enough free memory there shoud occur no paging. The problem is that the windows memory manager and my application are two autonomous processes. In good conditions everything is working fine, but the memory manager obviously is too slow in conditions where I'm entering low memory conditions, the memory is full and then the system starts to page (which is good), but I'm still allocating memory, because I don't get any information about the paging.
In the end I'm entering a state where the system stalls, the memory manager pages and I'm allocating.
So I came to the point where I need to advice the memory manager, check current memory conditions and invoke the paging myself. For that reason I wanted to use the GetWriteWatch to inspect the memory region I can flush.
Interestingly the GetWriteWatch does not work in my situation, it returns a -1 without filling the structures. So my question is does GetWriteWatch work with MMFs?
Does GetWriteWatch work with Memory-Mapped Files?
I don't think so.
GetWriteWatch accepts memory allocated via VirtualAlloc function using MEM_WRITE_WATCH.
File mapping are mapped using MapViewOfFile* functions that do not have this flag.
From MSDN i understood that when you invoke WinApi CreateFile without FILE_FLAG_NO_BUFFERING then WriteFile writes data to a system cache. It does not directly write to the physical disk. The OS will write it to the physical disk later on.
I wonder what happens when MoveFile is invoked. Is it guaranteed that after the MoveFile invocation the file is actually moved on the physical disk or is only the system cache updated?
If you want a guarantee that the move has made it to disk, use MoveFileEx with the MOVEFILE_WRITE_THROUGHflag, which does exactly that. Do note that this is possibly a serious performance impairment (with usually little or not benefit).
MoveFile by itself does not specify how the file is moved. It might indeed move, or it might copy-and-delete, and it might or might not use the buffer cache.
It is reasonable to assume that it indeed works using the buffer cache, and that "move" really means "move" on the same physical disk.
There is usually not much of a reason not to use the buffer cache, as apart from the computer crashing mid-operation or the user pulling the cable on an external disk, this is a perfectly reliable thing. Both scenarios are very rare. But even if they occur, the desastrous consequences are usually very mild, and very tolerable unless you tried to move huge directories with tens of thousands of files (usually, nothing was moved at all, or depending on the mode of operation, you have one intact original file and a stale file at the destination).
Is there a way for a process to walk it's own memory that is POSIX compliant? This must be done without modifying the process's source code, but we can interpose on system calls.
The application is for real-time systems which need to prevent unexpected delays from copy-on-write after a fork(). A solution is to call mlockall(), and then perform dummy writes to all memory that the process has mapped. In Linux, memory maps are available in /proc/self/maps (or /proc//maps), but this is not a requirement for POSIX compatibility.
In POSIX, I could only find posix_madvise(), which requires a given memory region as input. Also, posix_madvise() does not specify if it will affect the performance when writing and seems to work only on memory mapped files.
I believe this is a simple matter of building yourself a list of allocated data and then performing the writes. I've also seen recommendation to allocate a large dummy stack and fill it with zeros, so that stack variable access won't block. If I understood the problem, you don't need and should not copy program code.
With POSIX you won't be able to find and load pages that were allocated without program control.
I'm passing 1-2 MB of data from one process to another, using a plain old file. Is it significantly slower than going through RAM entirely?
Before answering yes, please keep in mind that in modern Linux at least, when writing a file it is actually written to RAM, and then a daemon syncs the data to disk from time to time. So in that way, if process A writes a 1-2 MB into a file, then process B reads them within 1-2 seconds, process B would simply read the cached memory. It gets even better than that, because in Linux, there is a grace period of a few seconds before a new file is written to the hard disk, so if the file is deleted, it's not written at all to the hard disk. This makes passing data through files as fast as passing them through RAM.
Now that is Linux, is it so in Windows?
Edit: Just to lay out some assumptions:
The OS is reasonably new - Windows XP or newer for desktops, Windows Server 2003 or newer for servers.
The file is significantly smaller than available RAM - let's say less than 1% of available RAM.
The file is read and deleted a few seconds after it has been written.
When you read or write to a file Windows will often keep some or all of the file resident in memory (in the Standby List). So that if it is needed again, it is just a soft-page fault to map it into the processes' memory space.
The algorithm for what pages of a file will be kept around (and for how long) isn't publicly documented. So the short answer is that if you are lucky some or all of it may still be in memory. You can use the SysInternals tool VMmap to see what of your file is still in memory during testing.
If you want to increase your chances of the data remaining resident, then you should use Memory Mapped Files to pass the data between the two processes.
Good reading on Windows memory management:
Mysteries of Windows Memory Management Revealed
You can use FILE_ATTRIBUTE_TEMPORARY to hint that this data is never needed on disk:
A file that is being used for temporary storage. File systems avoid writing data back to mass storage if sufficient cache memory is available, because typically, an application deletes a temporary file after the handle is closed. In that scenario, the system can entirely avoid writing the data. Otherwise, the data is written after the handle is closed.
(i.e. you need use that flag with CreateFile, and DeleteFile immediately after closing that handle).
But even if the file remains cached, you still have to copy it twice: from your process A to the cache (the WriteFile call), and from cache to the proces B (ReadFile call).
Using memory mapped files (MMF, as josh poley already suggested) has the primary advantage of avoiding one copy: the same physical memory pages are mapped into both processes.
A MMF can be backed by virtual memory, which means basically that it always stays in memory unless swapping becomes necessary.
The major downside is that you can't easily grow the memory mapping to changing demands, you are stuck with the initial size.
Whether that matters for an 1-2 MB data transfer depends mostly on how you acquire and what you do with the data, in many scenarios the additional copy doesn't really matter.
The function CreateFileMapping can be used to allocate space in the pagefile (if the first argument is INVALID_HANDLE_VALUE). The allocated space can later be memory mapped into the process virtual address space.
Why would I want to do this instead of using just VirtualAlloc?
It seems that both functions do almost the same thing. Memory allocated by VirtualAlloc may at some point be pushed out to the pagefile. Why should I need an API that specifically requests that my pages be allocated there in the first instance? Why should I care where my private pages live?
Is it just a hint to the OS about my expected memory usage patterns? (Ie, the former is a hint to swap out those pages more aggressively.)
Or is it simply a convenience method when working with very large datasets on 32-bit processes? (Ie, I can use CreateFileMapping to make >4Gb allocations, then memory map smaller chunks of the space as needed. Using the pagefile saves me the work of manually managing my own set of files to "swap" to.)
PS. This question is sparked by an article I read recently: http://blogs.technet.com/markrussinovich/archive/2008/11/17/3155406.aspx
From the CreateFileMappingFunction:
A single file mapping object can be shared by multiple processes.
Can the Virtual memory be shared across multiple processes?
One reason is to share memory among different processes. Different processes by only knowing the name of the mapping object can communicate over page file. This is preferable over creating a real file and doing the communications. Of course there may be other use cases. You can refer to Using a File Mapping for IPC at MSDN for more information.