I've always known that main memory caches data read from files on disk, but I have trouble reconciling this with the typical memory usage of a program, as shown in this diagram:
If I use fread or fgetl on a very large file 2010Stats.txt, on which segment of the above diagram does memory cache data from disk?
That's the address-space of your process. Each process has its own virtual memory address-space. Unless you mmap a file, its data only appears in your process's memory when copied there by a read system call or equivalent. (C stdio / iostream functions will be implemented on top of system calls like that.)
The disk-cache is managed by the OS kernel, which has to keep track of how the physical pages of memory are used, as well the virtual address mappings of every process.
Related
When the operating system loads Program onto the main memory , it , along with the stack and heap memory , also attaches the static data along with it. I googled about what is present in the static data which said it contained the global variables and static variables. But I am confused as both of these are already present in the text file of the program then why do we add them seperately?
The data in the executable is often referred as the data segment. The CPU doesn't interact with the hard-disk but only with RAM. The data segment must thus be loaded in RAM before the CPU can access it. The file of the executable is not really a text file. It is an executable so it has a different extension. Text files often refer to an actual file with a .txt extension.
With that said, you also asked another question not long ago (If the amount of stack memory provided to a program is fixed then why does it grow downwards in the process architecture? Or am I getting it wrong?) so I will try to give some insight for both of these in this same answer.
I don't know much about caching and low level inner CPU workings but, today mostly, the CPU doesn't even operate on RAM directly. It will load a bunch of RAM chunks into the cache and make operations on them and keep RAM-cache consistency by implementing complex mechanisms. The OS also has its role to play in RAM-cache consistency but, like I said, I am far from an expert here. Other than that, caching is mostly transparent to the OS. The CPU handles it and the OS simply provides instructions to the CPU which executes them.
Today, you have paging used by most OS and implemented on most CPU architectures. With paging, every process sees a full contiguous virtual address space. The virtual address space is accessed contiguously and the hardware MMU translates those addresses to physical ones automatically by crossing the page tables. The OS is responsible to make sure the page tables are consistent and the MMU does the rest of the job (for more info read: What is paging exactly? OSDEV). If you understand paging well, things become much clearer.
For a process, there is mostly 3 types of memory. There is the stack (often called automatic storage), the heap and the static/global data. I will attempt to give precision on all of these to give a global picture.
The stack is given a maximum size when the process begins. The OS handles that and creates the page tables and places the proper address in the stack pointer register so that stack accesses reach the proper region of physical memory. The stack is automatic storage which means that it isn't handled manually by the high level programmer. For example, in C/C++, the stack is managed by the compiler which, at the entry of a function, will create a stack frame and place offsets from the stack base pointer in the instructions. Every local variable (within a function) will be accessed with a relative negative offset from the stack base pointer. What the compiler needs to do is to create a stack frame of the proper size so that there will be enough place for all local variables of a particular function (for more info on the stack see: Each program allocates a fixed stack size? Who defines the amount of stack memory for each application running?).
For the heap, the OS reserves a very big amount of virtual memory. Today, virtual memory is very big (2^48 bytes or more). The amount of heap available for each process is often only limited by the amount of physical memory available to back virtual memory allocations. For example, a process could use malloc() to allocate 4KB of memory in C. The OS will be called with a system call by the libc library which is an implementation of the C standard library. The OS will then reserve a page of the virtual memory available for the heap and change the page tables so that accessing that portion of virtual memory will translate to somewhere in RAM (probably somewhere another process wasn't already using).
The static/global data are simply placed in the executable in the data segment. The data segment is loaded in the virtual memory alongside the text segment. The text segment will thus be able to access this data often using RIP-relative addressing.
outI'm working with memory mapped files (MMF) with very large datasets (depending on the input file), where each file has ~50GB and there are around 40 files open at the same time. Of course this depends, I can also have smaller files, but I can also have larger files - so the system should scale itself.
The MMF is acting as a backing buffer, so as long as I have enough free memory there shoud occur no paging. The problem is that the windows memory manager and my application are two autonomous processes. In good conditions everything is working fine, but the memory manager obviously is too slow in conditions where I'm entering low memory conditions, the memory is full and then the system starts to page (which is good), but I'm still allocating memory, because I don't get any information about the paging.
In the end I'm entering a state where the system stalls, the memory manager pages and I'm allocating.
So I came to the point where I need to advice the memory manager, check current memory conditions and invoke the paging myself. For that reason I wanted to use the GetWriteWatch to inspect the memory region I can flush.
Interestingly the GetWriteWatch does not work in my situation, it returns a -1 without filling the structures. So my question is does GetWriteWatch work with MMFs?
Does GetWriteWatch work with Memory-Mapped Files?
I don't think so.
GetWriteWatch accepts memory allocated via VirtualAlloc function using MEM_WRITE_WATCH.
File mapping are mapped using MapViewOfFile* functions that do not have this flag.
General malloc and mmap description
malloc (or any allocation function) is supposed to allocate memory for applications. Standard glibc malloc implementation uses sbrk() system call to allocate the memory. The memory allocated to an application is not backup by disk. Only when the application is swept out, the contents of memory are moved to disk (pre-configured swap disk).
The other method to allocate the memory is through the use of mmap. mmap system call creates mapping in virtual address space for calling process. Following is mmap function declaration as per POSIX standard.
void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset);
/* Describe the meaning for few important parameters to mmap */
mmap system call can also be used to allocate memory. Typically this is used to load, application binaries or static libraries. For example following mmap call will allocate memory, without a backing file.
address = __mmap (0, length, PROT_READ|PROT_WRITE,MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
Flags
MAP_ANONYMOUS: The mapping is not backed by any file; its contents
are initialized to zero.
MAP_PRIVATE: Create a private copy-on-write mapping. Updates to the mapping are not visible to other processes mapping the same file, and are not carried through to the underlying file.
dmalloc
dmalloc is a new API which allocates memory using a disk backed file i.e. without MAP_ANONYMOUS and MAP_PRIVATE to mmap. The dmalloc would be particularly useful with SSDs, which has very low read/write latency as compared to HDD. Since the file is mapped into the RAM, the dlmalloc will also benefit from high speed RAM.
Alternatives
A SSD can also be configured as a highest priority swap device, however this approach suffers from HDD optimized swapping algorithm inside Linux kernel. The swapping algorithm tries to cluster application pages on swap. When a data from swap is needed it reads
the complete cluster (read-ahead). If an application is doing random IOs, the read-ahead data would cause unnecessary IOs to disk.
Question:-
what is ment by "allocates memory using a disk backed file i.e. without MAP_ANONYMOUS and MAP_PRIVATE to mmap." which flag i should use apart from those two.
how i creat on-write backup of memory allocated to an application.
I never heard from dmalloc but like you mention it it looks like a mix between malloc (pure memory allocation) and mmap (direct mapping of memory to disk). dmalloc seems to allocate memory backed by disk, but more performant than mmap on slow disks (e.g. SSD). I could imagine that it groups write operations before actually flushing the writes to disk, whereas mmap is more or less a "virtual memory window" on a disk file.
As for your questions.
1) MAP_ANONYMOUS and MAP_PRIVATE are flags for use with mmap. The fact that these flags are mentioned as not being used, makes me think dmalloc is a fresh implementation and has no relationship to mmap.
2) dmalloc seems to be suited for what you say: it "backs up" memory to disk, similar to mmap. You may need to read the details of the documentation to know when exactly you have guarantee that the data is effectively on disk (caching,...)
I use boost::interprocess::managed_shared_memory to load a data structure in shared memory. I need to pin the shared memory segment into physical memory (for example similar to system call mlock for mapped files).
In linux, sooner or later my data structure gets swapped out of physical memory. In my case this incurs a forbidding cost for the next process accessing the structure, after it has been swapped out.
Is there any way to pin shared memory into physical memory? I am interested in any solution, even if it means that I cannot use boost::interprocess.
Using basic_managed_xsi_shared_memory (apparently available since boost 1.46), you can access the underlying shmid (from the get_shmid member) which should allow you to control the shmid using shmctl. With shmctl you can prevent the swapping of shared memory pages by applying the SHM_LOCK command to the shmid.
Other types of locking (which you refer to as 'pinning'), such as locking memory mapped files into memory, can be realized by supplying return values obtained from mapped_region's get_address and get_size member functions to the mlock command.
I was wondering whether there is any reasonably portable way to take existing, unshared heap memory and to convert it into shared memory. The usage case is a block of memory which is too large for me to want to copy it unnecessarily (i.e. into a new shared memory segment) and which is allocated by routines out of my control. The primary target is *nix/POSIX, but I would also be interested to know if it can be done on Windows.
Many *nixe have Plan-9's procfs which allows to open read a process's memory by inspecting /proc/{pid}/mem
http://en.wikipedia.org/wiki/Procfs
You tell the other process your pid, size and the base address and it can simply read the data (or mmap the region in its own address space)
EDIT:: Apparently you can't open /proc/{pid}/mem without a prior ptrace(), so this is basically worthless.
Under most *nixes, ptrace(2) allows to attach to a process and read its memory.
The ptrace method does not work on OSX, you need more magic there:
http://www.voidzone.org/readprocessmemory-and-writeprocessmemory-on-os-x/
What you need under Windows is the function ReadProcessMemory
.
Googling "What is ReadProcessMemory for $OSNAME" seems to return comprehensive result sets.
You can try using mmap() with MAP_FIXED flag and address of your unshared memory (allocated from heap). However, if you provide mmap() with own pointer then it is constrained to be aligned and sized according to the size of memory page (may be requested by sysconf()) since mapping is only available for the whole page.