If I see the output of cat /proc//smaps, I find that there are some memory regions with which no read/write/execute permissions have been associated. Also these region are mapped to inode number 0.
I wanted to know how does a region end up in such a state? Is it some sort of memory leak?
Can these regions be ever used again by the process?
They're not leaks. They're created by calling mmap() with the MAP_ANONYMOUS and PROT_NONE flags. The process can still use that virtual address space: it could unmap the regions with munmap(), or alter the protections with mprotect().
Such regions are typically used to set up guard pages, which are intended to trigger a signal when a growing data structure grows beyond its current bound.
Related
When you create EBPF maps, memory is allocated in kernel space. And kernel memory never gets swapped out. Then, why is there a need for the userspace application to call setrlimit() with RLIMIT_MEMLOCK?
RLIMIT_MEMLOCK refers to the amount of memory that may be locked into RAM, it is not specific to allocations in the user space memory address range. On pre-5.11 kernels, memory used for eBPF objects (programs, maps, BTF objects etc.) is accounted against this resource, meaning that if you create too many of them at a given time, or even in a short interval of time (given that there is a small delay after the deletion of an object before the kernel reclaims the corresponding memory area), you may hit the resource limit and get a -EPERM as an answer. For privileged users, calling setrlimit() to lift the restrictions on this resource is the usual workaround indeed.
Note that in Linux 5.11, rlimit-based accounting was dropped in favour of cgroup-based memory accounting. The rlimit had a number of downsides (refer to the cover letter linked above for details), and the cgroup-based accounting is more flexible, while allowing a better control and offering easier ways to retrieve the current amount of memory used. It should also reflect the actual memory consumption, while this was not necessarily the case with rlimit.
I started searching and reading about ALDS and memory management recently after I got a doubt about memory allocation, and after a couple of days of study I learnt a lot of things about memory management but the actual doubt remains unsolved.
So the doubt is, while allocating memory to a variable, how exactly does the system know which block of memory is available and which is free, and similarly when we destruct an object or set a variable as null or when GC frees up some memory, what exactly does it do with that block of memory, as I know the actual data is never erased on deletion, that block just gets marked as free somewhere in some table, but does that table keep track of each and every bit on the memory, if yes then wouldn't that become a lot of data in itself to store?
For an example, if I declare a linked list, then a block will be allocated in heap with it's next block having null value as there is no other node to reference, now as I keep adding more nodes into it, system will keep allocating more blocks each containing reference to next one. Now these blocks can be present on random locations depending on the availability of memory at allocation time, and can only be accessed through their proceeding nodes.
So now, for any given block of memory, how the system will know if its free and has just garbage value in it, or its actually a node of some linked list.
On a modern operating system the process has a logical, linear address space. Part of that address space is reserved for the system and is common to all processes. Some of the address space may be reserved but most of the remainder is available to the process.
The address space is defined by PAGE TABLES. The structure of the page table is defined by the processor but the operating system maintains a table for each process. Memory is allocated to a process in PAGES. The smallest I am aware of is 512 bytes but the size can go up to a megabyte or even larger in some processors and some processor configurations.The size is always a power of 2.
The page table defines:
Whether an page has actually been mapped to the process
Whether the pages has a corresponding physical memory location
If so, the mapping to that physical location.
There operating system only knows about pages.
At the next level down there are memory managers. These are not part of the operating system. Memory managers manage heaps that consist of pages allocated by the operating system. The memory manage has to keep track of the heap size and what memory has been allocated within it.
Memory managers operate is a huge number of different ways. There are malloc/free implementations galore that you can link into your code to get different behaviors.
I was reading OS from Galvin and just had a doubt, how to implement memory protection if the system does not supports virtual memory ? I mean how processes can be given protected address spaces ?
Any new concept or explaination would be awesome...
Memory protection on Wikipedia shows different methods of memory protection, you should go through that.
If there is no support of virtual-memory, the concept of Protection keys method can be used.
Reading it :
A memory protection key (MPK) mechanism divides physical memory up
into blocks of a particular size (e.g., 4 kiB), each of which has an
associated numerical value called a protection key. Each process also
has a protection key value associated with it. On a memory access the
hardware checks that the current process's protection key matches the
value associated with the memory block being accessed; if not, an
exception occurs.
If you want to understand operating systems, use Galvin to line your cat box and nothing else.
The first key to understanding this problem is to understand the different between logical memory and virtual memory. Sadly, much process and OS documentation conflates the two terms.
Page tables define a continuous, linear, logical address space that maps pages to non-continuous page frames. => Logical memory.
Virtual memory is the process of a using secondary storage (disk) when a page table entry does not map to a physical page frame.
These are two closely related concepts. Virtual memory requires logical memory BUT logical memory does not require virtual memory.
In the latter case, the OS does not use secondary storage to simulate physical memory. When a process accesses a logical page, it must be mapped to a physical page frame, otherwise there is an error (and not a page fault, as in virtual memory).
In the pre-virtual memory days, things were done entirely differently. Then entire processes were swapped in and out (why in Eunuchs, the page file is called the swap partition).This was done using read/write to disk or mapping registers. In those days the address spaces were 64K bytes or smaller so writing a process to disk was not as onerous as it might seem.
In such systems, you got your own 64K or so memory and simply could not access beyond that to get to another process's memory.
I have come across this question:
If process A contains a pointer to a variable in process B, is it
possible for A to access and modify that variable?
My intuition is that, since processes A and B are different, they should not be allowed to access each other's address space since it will violate the protection.
But after some thinking, the following questions popped in my mind and want to get clarified.
(i). When we say A has a pointer to a variable V in B, does A holds the virtual address (of process B) corresponding to V or the physical address?
I believe when we talk about address in virtual memory systems, we always talk about virtual address. Please clarify.
(ii). If A contains the virtual address, since it is possible that both A and B can have the same virtual address, it is possible that A's pagetable contains a mapping for the virtual address that A holds (which is actually the virtual address of variable V in process B).
Then when A tries to access and modify that virtual address, it modifies something in its own address space (this access will be allowed since A accesses its own address).
I think the above reasoning applies when we try to access some random virtual address from a process i.e., accidentally the address that we try to access has a valid mapping.
Please throw your thoughts.
The whole point of processes and memory management in the form they appear in modern OS's is that you cannot have a pointer from one process to another. Their memory is separated and one process cannot usually see the memory of another memory. To each process it looks like it has almost all the memory of the system available to it, as if there were only this one process (and the kernel, which might map stuff into the process' memory region).
The exception is shared memory: if both processes share a shared memory region and both processes have the access rights to modify the region, then yes, one process can modify the memory of the other process (but only within the bounds of that shared memory region).
IIRC, it works like this on the lowest level: the kernel manages a list of memory regions for each process. These regions might map to physical memory. If a region isn't mapped to physical memory and the process tries to access the region, the CPU signals the kernel to make it available (for example by loading its content from a swap file/partition). If two processes use shared memory, for both processes these regions would map to the same physical memory location (or swap file location). You might want to read about MMU and virtual memory.
You're exactly right. This is the way that virtual memory works. However, memory can be shared between processes.
For example, mmap in Linux can be used to create a shared mapping that will allow 2 seperate processes to access the same memory. I'm not sure if this works by mapping the virtual addresses for the 2 processes to the same piece of physical memory of by the same technique that memory mapped I/O works (pages are marked "dirty", then the operating system is responsible for doing the actual I/O), but from the point of view of the programmer, it's exactly as if 2 threads were accessing that memory.
In addition to what everyone said: all modern OSes have mechanisms for debugging, and that requires one process (the debugger) to access the memory of another (the debuggee). In Windows for example, there are APIs ReadProcessMemory()/WriteProcessMemory(), but there's a privilege barrier to their use.
Yes, there's some abuse potential. But how would you debug otherwise?
I was wondering whether there is any reasonably portable way to take existing, unshared heap memory and to convert it into shared memory. The usage case is a block of memory which is too large for me to want to copy it unnecessarily (i.e. into a new shared memory segment) and which is allocated by routines out of my control. The primary target is *nix/POSIX, but I would also be interested to know if it can be done on Windows.
Many *nixe have Plan-9's procfs which allows to open read a process's memory by inspecting /proc/{pid}/mem
http://en.wikipedia.org/wiki/Procfs
You tell the other process your pid, size and the base address and it can simply read the data (or mmap the region in its own address space)
EDIT:: Apparently you can't open /proc/{pid}/mem without a prior ptrace(), so this is basically worthless.
Under most *nixes, ptrace(2) allows to attach to a process and read its memory.
The ptrace method does not work on OSX, you need more magic there:
http://www.voidzone.org/readprocessmemory-and-writeprocessmemory-on-os-x/
What you need under Windows is the function ReadProcessMemory
.
Googling "What is ReadProcessMemory for $OSNAME" seems to return comprehensive result sets.
You can try using mmap() with MAP_FIXED flag and address of your unshared memory (allocated from heap). However, if you provide mmap() with own pointer then it is constrained to be aligned and sized according to the size of memory page (may be requested by sysconf()) since mapping is only available for the whole page.