As I understand the creation of processes, every process has it's own space in RAM for it's heap, data, etc, which is allocated upon its creation. Many processes can share their data and storage space in some ways. But since terminating a process would erase its allocated memory(so also its caches), I was wondering if it is possible that many (similar) processes share a cache in memory that is not allocated to any specific process, so that it can be used even when these processes are terminated and other ones are created.
This is a theoretical question from a student perspective, so I am merely interested in the general sence of an operating system, without adding more functionality to them to achieve it.
For example I think of a webserver that uses only single-threaded processes (maybe due to lack of multi-threading support), so that most of the processes created do similar jobs, like retrieving a certain page.
There are a least four ways what you describe can occur.
First, the system address space is shared by all processes. The Operating system can save data there that survives the death of a process.
Second, processes can map logical pages to the same physical page frame. The termination of one process does not cause the page frame to be deallocated to the other processes.
Third, some operating systems have support for writable shared libraries.
Fourth, memory mapped files.
There are probably others as well.
I think so, when a process is terminated the RAM clears it. However your right as things such as webpages will be stored in the Cache for when there re-called. For example -
You open Google and then go to another tab and close the open Google page, when you next go to Google it loads faster.
However, what I think your saying is if the Entire program E.G - Google Chrome or Safari - is closed, does the webpage you just had open stay in the cache? No, when the program is closed all its relative data is also terminated in order to fully close the program.
I guess this page has some info on it -
https://www.wikipedia.org/wiki/Shared_memory
Related
I am creating a Golang program that creates a process and then should be able to suspend it.
To make it more memory efficient, I would need my program to be able to dump the memory of the process to disk and reload it only when needed.
I cannot find any info here on Stack Overflow and also GitHub is not helping.
Any solution?
Attempting to answer this with the limited info..
To make it more memory efficient, I would need my program to be able to dump the memory of the process to disk and reload it only when needed.
This is generally something handled by your operating system (scheduler, memory management) controlling what processes are currently running / suspended / etc. and what memory needs to be paged in / out. Trying to implement the equivalent is quite complex, error prone, and likely to be less performant. Why do you believe you need to implement this yourself?
If you are building a program and want to have explicit control about whether it should be considered runnable or not, you could create a process which forks (creating two total processes), and have the parent process suspend and resume the child process using signals:
https://man7.org/linux/man-pages/man7/signal.7.html
As the title says, I want to associate a random bit of data (ULONG) with a running process on the local machine. I want that data persisted with the process it's associated with, not the process thats reading & writing the data. Is this possible in Win32?
Yes but it can be tricky. You can't access an arbitrary memory address of another process and you can't count on shared memory because you want to do it with an arbitrary process.
The tricky way
What you can do is to create a window (with a special and known name) inside the process you want to decorate. See the end of the post for an alternative solution without windows.
First of all you have to get a handle to the process with OpenProcess.
Allocate memory with VirtualAllocEx in the other process to hold a short method that will create a (hidden) window with a special known name.
Copy that function from your own code with WriteProcessMemory.
Execute it with CreateRemoteThread.
Now you need a way to identify and read back this memory from another process other than the one that created that. For this you simply can find the window with that known name and you have your holder for a small chunk of data.
Please note that this technique may be used to inject code in another process so some Antivirus may warn about it.
Final notes
If Address Space Randomization is disabled you may not need to inject code in the process memory, you can call CreateRemoteThread with the address of a Windows kernel function with the same parameters (for example LoadLibrary). You can't do this with native applications (not linked to kernel32.dll).
You can't inject into system processes unless you have debug privileges for your process (with AdjustTokenPrivileges).
As alternative to the fake window you may create a suspended thread with a local variable, a TLS or stack entry used as data chunk. To find this thread you have to give it a name using, for example, this (but it's seldom applicable).
The naive way
A poor man solution (but probably much more easy to implement and somehow even more robust) can be to use ADS to hide a small data file for each process you want to monitor (of course an ADS associated with its image then it's not applicable for services and rundll'ed processes unless you make it much more complicated).
Iterate all processes and for each one create an ADS with a known name (and the process ID).
Inside it you have to store the system startup time and all the data you need.
To read back that informations:
Iterate all processes and check for that ADS, read it and compare the system startup time (if they mismatch then it means you found a widow ADS and it should be deleted.
Of course you have to take care of these widows so periodically you may need to check for them. Of course you can avoid this storing ALL these small chunk of data into a well-known location, your "reader" may check them all each time, deleting files no longer associated to a running process.
I was trying to understand following:
I know that page tables are built for translation between virtual memory and physical memory by virtual memory manager at some point. Since there are many processes running on a system, even though only process active at a time, I was wondering whether page tables for inactive process are moved to page file at any point of time? Given the fact that lower 2 GB area is reserved for windows, it would make sense that windows would keep page tables for all processes on the system. Although it would make sense as well that they are moved to page file if the current process is switched?
Same goes for the writable (data) pages. Will windows keep all the data pages for all the process in memory or move them to page file at some point. On my machine, task manager says 1.5 GB RAM is being utilized out of 3 GB and 1.5 is system cache in performance tab so my understanding is data stays in physical memory for all applications. But would there be a time when it needs to moved to paging file?
I was wondering whether page tables for inactive process are moved to page file at any point of time?
Yes, page tables are pageable.
Will windows keep all the data pages for all the process in memory or move them to page file at some point.
As far as the Windows paging policy is concerned, there's two kinds of memory: pageable and non-pageable. It doesn't really matter which process it belongs to or even if it belongs to the O/S itself, if it's pageable then it's subject to being paged out. So, yes, Windows will page out process data pages if necessary.
I suggest reading the memory management chapter in the Windows Internals book, it should cover all of this.
-scott
You are actually asking two questions here.
What's the paging policy regarding the page tables.
What's the paging policy for "writable data" pages (i.e. virtual memory with R/W permissions).
First I'll correct you a little.
Given the fact that lower 2 GB area is reserved for windows, it would
make sense that windows would keep page tables for all processes on
the system
To be exact it's the upper 2GB that are reserved to windows, more correctly - may be accessed in the kernel mode only by Windows kernel and drivers.
Now, this may surprise you, but the kernel memory may be pagable too! So technically it's not important at all which portion of the 32-bit address space is visible in the user/kernel mode. It's not related to paging.
Another correction: virtual memory may be in physical memory and saved to the page file. There's a common belief that the OS frees physical storage by on-demand saving the pages to the page file. Wrong.
Actually Windows saves memory pages to the page file before they need to be freed. In fact it dumps all the memory pages to the page file (besides of those that are related to other files, such as mapped sections) in background. There are two reasons for this:
During high load the OS will free memory pages quicker (since they're already saved)
In the kernel mode paging is not always possible. Drivers that run on high IRQL (i.e. serve the most time-critical events) may not access physical storage drivers, hence paging is not possible.
So, the answers to your questions are:
Don't know for sure, but it depends on the OS implementation details. I see no reasons why per-process page table may not be paged-out. It's needed during the context switch and modifying process virtual memory. Both situations don't belong to the time-critical events.
Definitely "writable data" memory pages are saved to the page file. Are they removed from the physical memory? On-demand only, during the system load, in the least-recent-used order.
I've been tracking down a few memory leaks in my application. It's been a real pain, but I've finally tightened everything up. However, there's one bit of Windows memory management that is confusing me. Here is a printout of the app's memory usage over time...
Time PrivateMemorySize64 WorkingSet64
20:00:36 47480, 50144
20:01:06 47480, 50144
20:01:36 47480, 50144
20:02:06 47480, 149540
20:02:36 47480, 149540
20:03:06 47480, 149540
The working set jumped from 49 MB to 146 over a span of 30 seconds. This happened overnight as the application was basically doing nothing.
The working set (which is what task manager shows me) seems to be able to be influenced by other applications such as debuggers (as I learned while looking for memory leaks). After reading the documentation on what Working Set is, I still don't have a good understanding.
Any help is appreciated.
Update: Thanks to some links from responders as well as some additional searching, I have a better understanding on how a separate process can cause my process' working set to grow. Good to know that a spike in the working set is not necessarily an indication that your app is leaking... Further reason to not rely on Task Manager for memory evaluation :)
Helpful links:
A few words on memory usage or: working set vs. private working set
CyberNotes: Windows Memory Usage Explained
Simply said, the working set is the collection of memory pages currently owned by your process and not swapped out (i.e. in RAM). That is somewhat inaccurate, however. Reality is a lot more complicated.
Windows maintains a minimum working set size and a maximum working set size for every process. The minimum working set is easy, it is what Windows will grant to every process (as long as it can, by physical limits).
The maximum working set is more dubious. If your program uses more memory than will fit in its quota, Windows will drop some pages. However, while they are no longer in your working set, these pages are not necessarily "gone".
Rather, those pages are removed from your working set and moved to the pool of available pages. As a consequence, if some other program needs more memory and no cleared pages are left over, your pages will be cleared, and assigned to a different process. When you access them, they will need to be fetched from the swapfile again, possibly purging other pages, if you are still above the maximum working set size.
However, if nobody asked for more memory in the mean time (or if all demands could be satisfied by pages that were unused anyway), then accessing one of those pages will simply make it "magically reappear" and kick out another page in its stead.
Thus, your process can have more pages in RAM than are actually in its working set, but it does not "officially" own them.
The Resident Set/Working Set is the portion of Virtual Address Space which is currently residing in Physical Memory and therefore isn't Swapped Out
Are you meant to protect against simultanously reads/writes to file mapped memory that is open by multiple processes?
For example if a string in the memory is "hello" and one process writes "hi..." over it, am I correct to say that another process that reads at the same time may get an intermittant value like "hi.lo"?
Basically what I am asking is how do people protect again these sorts of things. Are you meant to use semaphores? Do these work across processes?
Yes, if you need to protect against multiple writers or avoid reading partial updates then a shared Mutex / Semaphore used by each process would work to control access to the shared data.
There is some sample code which does this at the bottom of this MSDN article: Memory-Mapped Files in .NET 4.0