I am reading the source code of mmap(), and I found the Linux kernel will make the files map into the vma struct. vma only represents memory areas. So, will the OS immediately allocate physical memory for the process after mmap()?
No, Linux may defer allocation of the physical memory, requested by mmap, until that memory will be accessed.
When application accesses an unmapped memory, page fault exception is triggered; when handle this exception, Linux may map that accessed memory, allowing the application to continue.
Related
To allocate memory dynamically we use some system calls . so the operating system allocates memory on the heap of corresponding process. but how is memory allocated in operating system itself?
how allocates operating system memory when some data structure (e.g process tabel) must be created? dose OS kernel has its own stack and heap?
The Linux operating system does use have its own fixed size stack for each thread. you can see the stack in the source code defining thread_union.
When a syscall is preformed in x86/x64 the kernel swaps out the GS segment and CR3 registers to refer to its own memory and then uses the GS segment register to locate cpu_current_top_of_stack which points to the corresponding kernel stack pointer for the calling thread. You can see this in entry_SYSCALL_64.
When the kernel needs to allocate dynamic memory it uses kamlloc which uses the slob allocator.
I've always known that main memory caches data read from files on disk, but I have trouble reconciling this with the typical memory usage of a program, as shown in this diagram:
If I use fread or fgetl on a very large file 2010Stats.txt, on which segment of the above diagram does memory cache data from disk?
That's the address-space of your process. Each process has its own virtual memory address-space. Unless you mmap a file, its data only appears in your process's memory when copied there by a read system call or equivalent. (C stdio / iostream functions will be implemented on top of system calls like that.)
The disk-cache is managed by the OS kernel, which has to keep track of how the physical pages of memory are used, as well the virtual address mappings of every process.
I have a Linux device driver that allocates physical memory/pages. I have a single ioctl, which any application can call, which simply allocates a physical page and maps it to process memory using vm_insert_pfn.
I have already allocated a contiguous VMA based on the total number of pages required by the application. I am using alloc_page() to allocate a physical page.
What I want to know is, if the physical page allocated by alloc_page() is counted against the process making the IOCTL or if it is simply a physical page that belongs to the driver even after mapping it to pre-allocated VMA. If not how can I achieve this?
I am using flags like GFP_MOVABLE|GFP_HIGHUSER.
I have an SoC which has both DSP and ARM cores on it and I would like to create a section of shared memory that both my userspace software, and DSP software are able to access. What would be the best way to allocate a buffer like this in Linux? Here is a little background, right now what I have is a kernel module in which I use kmalloc() to get a kernel buffer, I then use the __pa() macro from asm/page.h to get the physical address of my kernel buffer. I save this address as a sysfs entry so that my userspace code can get the physical address of this buffer. I can then write this address to the DSP so it knows where the shared memory location is, and I can also mmap /dev/mem or my own kernel module so that I can access this buffer from userspace (I could also use the read/write fileops).
For some reason I feel like this is overboard but I cannot find the best way to do what I am trying to do.
Would it be possible to just mmap \dev\mem a section of memory and just read and write to this section? My feeling is that this would not 'lock' this section of memory from the kernel, thus the kernel could still read/write to this memory without me knowing. Is this the case. After reading the memory management chapter of LDD3 I see that mmap creates a new VMA of the mapping. Would this lock this area of memory so that other processes would not get allocated this section of memory?
Any and all help is appreciated
Depending on the kind of DMA you're using, you need to allocate the buffer with dma_alloc_coherent(), or use standard allocations and the dma_map_* functions.
(You must not use __pa(); physical addresses are not necessarily the same as DMA bus addresses.)
To map the buffers to user space, use dma_mmap_coherent() for coherent buffers, or map the memory pages manually for streaming buffers.
For a similar requirement of mine, I had reserved about 16 MB of memory towards the end of ram and used it in both kernel and user space. Suppose you have 128 MB ram, you can set BOOTMEM argument as 112 MB in your boot loader. I am assuming you are using uboot. This will reserve 16 MB towards the end of the ram. Now in kernel and user space you can map this area and use it as shared memory.
I'm running a 32 bit system in legacy mode on a 64-bit (x86-64 that is) capable architecture. When a new process is created, the kernel has to decide where in physical memory all of the pages needed at the time of instantiation are to be allocated (assuming a single thread this may include several memory regions such as the stack, the heaps etc).
I'm assuming the kernel keeps some sort of dynamic list of the physical RAM frames that are in use, and also a static list of all the regions of physical memory that have been taken up by devices for systems that use memory-mapped IO. Is this correct?
In addition, I also read that a 32-bit Windows system has a physical memory limit of 4GB (probably due to minimum address bus assumptions) so, even though a system may have more than 4 gigabytes of physical memory installed, a 32 bit kernel will only allocate addresses within the 4GB range.
Specific information regarding low-level operating system implementation for specific cases such as this is quite difficult to find online. Can anyone verify these statements and possibly refer me to a source where I could attain more information?
Thanks for your considerations.
When a new process is created, the kernel has to decide where in physical memory all of the pages needed at the time of instantiation are to be allocated
Why does it have to decide at process creation time? In fact, it only creates them on-demand - it simply creates the PTEs (i.e. "This address range is valid", but the pages are not backed in any way); when the process first starts executing, it immediately page-faults.
What is a page fault though? What happens is, first the CPU reads the TLB to see if it has an address <=> frame mapping. When that fails, it walks the PTEs looking for an entry that matches. If no entry is found, or if the entry indicates that the page isn't backed, a page-fault is generated. This means, that a CPU exception occurs and the CPU immediately jumps to a predefined address. The first thing the kernel then does is save the CPU Context (i.e. the registers at the location of the fault), then dispatches to the page fault handler.
When the page-fault occurs, Mm (the Memory Manager in NT) will read the mapping in its own data structures (remember that all PE images are memory-mapped files) and determine at that time which physical frame (i.e. 'a real piece of memory') which will be used.
Once the page fault is serviced, the page fault restores the saved CPU context, and jumps back to where it was, and retries the instruction that faulted.
You're correct that a 32-bit OS will only use 4GB of address space (not RAM! Don't forget those memory-mapped devices and files!), the processor will operate in 32-bit mode and interpret the PTEs as 32-bit (remember that AMD64 long mode adds an extra level of page tables and extends the address space to 48 bits).
32bit systems can only ever address 4gig directly (2^32 = 4gig). There's PAE hacks, which let the system have more than 4gig of physical ram, but no process can ever have more than 4gig available. As well, even if you have 4gig of ram, you'll never see more than 3.5gig or so actually available - some is reserved for memory mapping hardware devices, such as your video ram.
For one method of dealing with the physical-virtual memory mapping, look at TLB