Are they refering to non-pagable memory or what? I cannot find any reference to resident storage.
Resident memory, as mentioned here, here and here simply referes to memory that is not currently paged. Memory allocated from the NonPagedPool is for instance always resident while memory allocated from the PagedPool can be resident but you have to check that the memory hasn't been paged out before you use it (depending on the context IRQL requirements).
Related
In my research project, I have to reserve a large virtual memory address space inside a kernel module and handle memory accesses to that area (in a 64-bit system). I have modified do_page_fault function in arch/x86/mm/fault.c so that I can handle page faults when there is no page mapping record in TLB.
In my project, the reserved virtual address space is much larger than physically available memory. For example, let's say we have 4GB available memory space. Inside my kernel module, I what to have 16GB address space which is accessible by load/store instructions. I want to compress a 4KB page into 1KB and maintain 16GB compressible data into 4GB memory; once a page fault occurs, it'll decompress the corresponding page and copy it into the page cache.
I've tried memory allocate functions such as kmalloc and vmalloc, but they literally allocate memory while I only need some non-one-to-one virtual address mapping via calling a [decompression] function. How can I do that?
What is difference between kmem_cache_alloc and kmalloc() in kernel memory allocation? which one is used when?
Kmalloc - allocates contiguous region from the physical memory. But keep in mind, allocating and free'ing memory is a lot of work.
Kmem_cache_alloc - Here, your process keeps some copies of the some pre-defined size objects pre-allocated. Say you have struct that you know you will be requiring very frequently, so instead of allocating it from the main memory (kmalloc) when you need it, you already keep multiple copies of it allocated & when you want it, it returns the address of the block already allocated (saves a lot of time). Similarly, when you free it, you don't give it back, it actually isn't free'd, it goes back to the allocated pool so that if some process again asks for it, you can return this address of the already allocated struct.
kmalloc: It uses the generic slab caches available to any kernel code. so your module will share slab cache with other components in kernel.
kmem_cache_alloc: It will allocate objects from a dedicated slab cache created by kmem_cache_create. If you specifically want a better slab cache management dedicated to your module only, that too for a specific type of objects, use kmem_cache_create followed by kmem_cache_alloc. USB/SCSI drivers use this. kmem_cache_create takes sizeof your object you want to create slab of, a name which appears in /proc/slabinfo and flags to govern behavior of your slab cache.
Ref: https://www.mail-archive.com/kernelnewbies#nl.linux.org/msg13191.html & LDD
I trying to to make the linux memory management a little bit more clear for tuning and performances purposes.
By reading this very interesting redbook "Linux Performance and Tuning Guidelines" found on the IBM website I came across something I don't fully understand.
On 32-bit architectures such as the IA-32, the Linux kernel can directly address only the first gigabyte of physical memory (896 MB when considering the reserved range). Memory above the so-called ZONE_NORMAL must be mapped into the lower 1 GB. This mapping is completely transparent to applications, but allocating a memory page in ZONE_HIGHMEM causes a small performance degradation.
why the memory above 896 MB has to be mapped into the lower 1GB ?
Why there is an impact on performances by allocating a memory page in ZONE_HIGHMEM ?
what is the ZONE_HIGHMEM used for then ?
why a kernel that is able to recognize up to 4gb ( CONFIG_HIGHMEM=y ) can just use the first gigabyte ?
Thanks in advance
When a user process traps in to the kernel, the page tables are not changed. This means that one linear address space must be able to cover both the memory addresses available to the user process, and the memory addresses available to the kernel.
On IA-32, which allows a 4GB linear address space, usually the first 3GB of the linear address space are allocated to the user process, and the last 1GB of the linear address space is allocated to the kernel.
The kernel must use its 1GB range of addresses to be able to address any part of physical memory it needs to. Memory above 896MB is not "mapped into the low 1GB" - what happens is that physical memory below 896MB is assigned a permanent linear address in the kernel's part of the linear address space, whereas as memory above that limit must be assigned a temporary mapping in the remaining part of the linear address space.
There is no impact on performance when mapping a ZONE_HIGHMEM page into a userspace process - for a userspace process, all physical memory pages are equal. The impact on performance exists when the kernel needs to access a non-user page in ZONE_HIGHMEM - to do so, it must map it into the linear address space if it is not already mapped.
Are there any restrictions on memory usage by a Linux Kernel Module i.e Code Segment size or amount of global memory or any thing.
In 2.6.35, load_module() bails out if the length of the module to load exceeds 64 MB: http://lxr.linux.no/#linux+v2.6.35/kernel/module.c#L2118
vmalloc() is used to allocate space for the module -- this fails if you try to allocate more pages than available in your physical memory (which in turn will probably only be an issue for embedded stuff with low RAM)
Furthermore, kzalloc() (and in turn, kmalloc()) are used. Depending on the allocator used (SLAB, SLOB, SLUB), there may be restrictions as well. SLAB defines a KMALLOC_MAX_SIZE wich defines the maximum number of bytes you can allocate with a single call to kmalloc().
I understand that memory has to be reserved before being committed. And when it's reserved, no other process can use it. However reserved memory does not count against available RAM. But shouldn't it? Because if no one else can use it, then what good is it being "available"?
Or is there some bigger difference?
In the context of Win32, "reserved" means that the address space is allocated within the process that requested it. This may be used, for example, to reserve space for a large buffer that's not all needed right away, but when it's needed it would need to be contiguous.
Reserving memory does not interact with other processes at all, since each process has its own private address space. So the statement that "when it's reserved, no other process can use it" is meaningless, since processes can't normally allocate memory in the address space of another process anyway.
When the reserved pages are requested to be committed (backing store allocated for them), that operation can potentially fail due to lack of physical memory (or pagefile).
I like to view Reserved as booking the address space so that no one else can allocate it (but I can't use memory at that address because it is not yet available). And Committed as mapping that address space to the physical memory so that it can be used.
Why would I want to reserve? Why not just get committed memory? There are several reasons I have in mind:
Some application needs a specific address range, say from 0x400000 to 0x600000, but does not need the memory for storing anything. It is used to trap memory access. E.g., if some code accesses such area, it will be caught. (Useful for some reason.)
Some thread needs to store progressively expanding data. And the data needs to be in one contiguous chunk of memory. It is preferred not to commit large physical memory at one go because it is not needed and would be such a waste. The memory can be utilized by some other threads first. The physical memory is committed only on demand.
Process Virtual Memory (Address Space ) and Actual RAM are both different. you can have 512MB physical RAM but still your process can address 4GB virtual address space(2GB Userspace)
Every address in a process can be thought of as either free, reserved, or committed at any given time.
A process begins with all addresses free, meaning they are free to be committed
to memory or reserved for future use.Before any free address may be used, it must first be allocated as reserved OR committed. But doesn't need to be reserved in order for it to be committed.
reserving memory means reserving virtaul address space for future purposes. it doesn't associated with Physical RAM (mapped to RAM Addresses).where as committed memory means it will be associated with actual RAM so that you can store data in it.
http://msdn.microsoft.com/en-us/library/ms810627.aspx
I think the simplest answer is that:
you reserve memory in virtual address space, so that no other part of code within the same process gets it
you commit memory to physical RAM/swap so that no other process gets it
So for example if a process has mem limit of 1 GB and does malloc reserve of 1 GB at start it cannot malloc any more even though process memory usage at OS level (commited memory) is nearly 0.