I'm trying to reduce the amount of forking our Ruby/Rails app does. We shell out a lot with backticks, and each of these forks the entire process, which can cause a huge memory bloat.
I'm going through, identifying the ones that get called the most, and trying to replace them with code which achieves the same thing without making a shell call. However, in some cases I suspect it might still be forking under the hood anyway.
Is there a way to detect or log whenever a process forks? I'm using ubuntu 14.04. A log would be ideal as I can then keep an eye on it when I run the amended code.
I have a .NET project that I've been compiling with Visual Studio and running successfully on Windows for years. I'd like to move to use Mac OS X to run this particular program, and so have been working with mono for just a little while do make that possible. The results have been excellent, with the program performing as expected under mono (2.10.9, BTW).
The only hitch is that the program does not exit after it is done. It is a single-threaded process, but when its work is completed (which it reports at the end of the Main(string[]) method), the process does not exit; ^C is required to return to the shell prompt. Adding an explicit System.Environment.Exit(exitCode); call at the end has no effect.
I found only one other tale of this sort of problem from some years ago, here, though that ML thread provided no potential solution.
Rolf (in his comment on the question) was right, the vm was deadlocking upon shutdown.
I'm using IKVM in this application, and there are certain circumstances where it needs to start a thread at vm shutdown to clean up its JVM<->CLR interop machinery; this apparently does not sit well with mono (but always works in .NET).
This was discovered on a thread on the IKVM dev list by IKVM's author, who filed this bug with the mono project.
The workaround in my case was to explicitly call java.lang.System.exit(exitCode) instead of simply letting the program terminate naturally or using System.Environment.Exit(exitCode). This allows IKVM to shut down outside of the vm termination lifecycle, thereby avoiding the mono bug.
My problem is not the best scenario for fork(). However, this is the best func I can get.
I am working on a Firefox plugin on Mac OSX. To make it robust, I need to create a new process to run my plugin. The problem is, when I forked a new process, much like this:
if (fork() == 0) exit(other_main());
However, since the state is not cleaned, I cannot properly initialized my new process (call NSApplicationLoad etc.). Any ideas? BTW, I certainly don't want create a new binary and exec it.
In general, you need to exec() after fork() on Mac OS X.
From the fork(2) man page:
There are limits to what you can do in the child process. To be totally safe you should restrict your-self to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use these frameworks in the child process, you must exec. In this situation it is reasonable to exec yourself.
TN2083 also comments on this subject:
Many Mac OS X frameworks do not work reliably if you call fork but do not call exec. The only exception is the System framework and, even there, the POSIX standard places severe constraints on what you can do between a fork and an exec.
IMPORTANT: In fact, in Mac OS X 10.5 and later, Core Foundation will detect this situation and print the warning message shown in Listing 13.
Listing 13: Core Foundation complaining about fork-without-exec
The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.
fork without exec is basically entirely unsafe on OSX. You will end up with stale mach ports for example.
I'm writing the FreeWRL plugin for Firefox (Linux at the moment, Mac & Windows soon).
http://freewrl.sourceforge.net/
It's based on fork+exec to launch FreeWRL and swallow its window into Firefox.
You'll have to use a pipe to correctly handle the possible failure of fork+exec or the failure of your child process :
How to handle execvp(...) errors after fork()?
Cheers,
C
I noticed when a file is executed on Windows (.exe or .dll), it is locked and cannot be deleted, moved or modified.
Linux, on the other hand, does not lock executing files and you can delete, move, or modify them.
Why does Windows lock when Linux does not? Is there an advantage to locking?
Linux has a reference-count mechanism, so you can delete the file while it is executing, and it will continue to exist as long as some process (Which previously opened it) has an open handle for it. The directory entry for the file is removed when you delete it, so it cannot be opened any more, but processes already using this file can still use it. Once all processes using this file terminate, the file is deleted automatically.
Windows does not have this capability, so it is forced to lock the file until all processes executing from it have finished.
I believe that the Linux behavior is preferable. There are probably some deep architectural reasons, but the prime (and simple) reason I find most compelling is that in Windows, you sometimes cannot delete a file, you have no idea why, and all you know is that some process is keeping it in use. In Linux it never happens.
As far as I know, linux does lock executables when they're running -- however, it locks the inode. This means that you can delete the "file" but the inode is still on the filesystem, untouched and all you really deleted is a link.
Unix programs use this way of thinking about the filesystem all the time, create a temporary file, open it, delete the name. Your file still exists but the name is freed up for others to use and no one else can see it.
Linux does lock the files. If you try to overwrite a file that's executing you will get "ETXTBUSY" (Text file busy). You can however remove the file, and the kernel will delete the file when the last reference to it is removed. (If the machine wasn't cleanly shutdown, these files are the cause of the "Deleted inode had zero d-time" messages when the filesystem is checked, they weren't fully deleted, because a running process had a reference to them, and now they are.)
This has some major advantages, you can upgrade a process thats running, by deleting the executable, replacing it, then restarting the process. Even init can be upgraded like this, replace the executable, and send it a signal, and it'll re-exec() itself, without requiring a reboot. (THis is normally done automatically by your package management system as part of it's upgrade)
Under windows, replacing a file that's in use appears to be a major hassle, generally requiring a reboot to make sure no processes are running.
There can be some problems, such as if you have an extremely large logfile, and you remove it, but forget to tell the process that was logging to that file to reopen the file, it'll hold the reference, and you'll wonder why your disk didn't suddenly get a lot more free space.
You can also use this trick under linux for temporary files. open the file, delete it, then continue to use the file. When your process exits (for no matter what reason -- even power failure), the file will be deleted.
Programs like lsof and fuser (or just poking around in /proc//fd) can show you what processes have files open that no longer have a name.
I think linux / unix doesn't use the same locking mechanics because they are built from the ground up as a multi-user system - which would expect the possibility of multiple users using the same file, maybe even for different purposes.
Is there an advantage to locking? Well, it could possibly reduce the amount of pointers that the OS would have to manage, but now a days the amount of savings is pretty negligible. The biggest advantage I can think of to locking is this: you save some user-viewable ambiguity. If user a is running a binary file, and user b deletes it, then the actual file has to stick around until user A's process completes. Yet, if User B or any other users look on the file system for it, they won't be able to find it - but it will continue to take up space. Not really a huge concern to me.
I think largely it's more of a question on backwards compatibility with window's file systems.
I think you're too absolute about Windows. Normally, it doesn't allocate swap space for the code part of an executable. Instead, it keeps a lock on the excutable & DLLs. If discarded code pages are needed again, they're simply reloaded. But with /SWAPRUN, these pages are kept in swap. This is used for executables on CD or network drives. Hence, windows doesn't need to lock these files.
For .NET, look at Shadow Copy.
If executed code in a file should be locked or not is a design decision and MS simply decided to lock, because it has clear advantages in practice: That way you don't need to know which code in which version is used by which application. This is a major problem with Linux default behaviour, which is simply ignored by most people. If system wide libs are replaced, you can't easily know which apps use code of such libs, most of the times the best you can get is that the package manager knows some users of those libs and restarts them. But that only works for general and well know things like maybe Postgres and its libs or such. The more interesting scenarios are if you develop your own application against some 3rd party libs and those get replaced, because most of the times the package manager simply doesn't know your app. And that's not only a problem of native C code or such, it can happen with almost everything: Just use httpd with mod_perl and some Perl libs installed using a package manager and let the package manager update those Perl libs because of any reason. It won't restart your httpd, simply because it doesn't know the dependencies. There are plenty of examples like this one, simply because any file can potentially contain code in use in memory by any runtime, think of Java, Python and all such things.
So there's a good reason to have the opinion that locking files by default may be a good choice. You don't need to agree with that reasons, though.
So what did MS do? They simply created an API which gives the calling application the chance to decide if files should be locked or not, but they decided that the default value of this API is to provide an exclusive lock to the first calling application. Have a look at the API around CreateFile and its dwShareMode argument. That is the reason why you might not be able to delete files in use by some application, it simply doesn't care about your use case, used the default values and therefore got an exclusive lock by Windows for a file.
Please don't believe in people telling you something about Windows doesn't use ref counting on HANDLEs or doesn't support Hardlinks or such, that is completely wrong. Almost every API using HANDLEs documents its behaviour regarding ref counting and you can easily read in almost any article about NTFS that it in deed does support Hardlinks and always did. Since Windows Vista it has support for Symlinks as well and the Support for Hardlinks has been improved by providing APIs to read all of those for a given file etc.
Additionally, you may simply want to have a look at the structures used to describe a file in e.g. Ext4 compared to those of NTFS, which have a lot in common. Both work with the concept of extents, which separates data from attributes like file name, and inodes are pretty much just another name for an older, but similar concept of that. Even Wikipedia lists both file systems in its article.
There's really a lot of FUD around file locking in Windows compared to other OSs on the net, just like about defragmentation. Some of this FUD can be ruled out by simply reading a bit on the Wikipedia.
NT variants have the
openfiles
command, which will show which processes have handles on which files. It does, however, require enabling the system global flag 'maintain objects list'
openfiles /local /?
tells you how to do this, and also that a performance penalty is incurred by doing so.
Executables are progressively mapped to memory when run. What that means is that portions of the executable are loaded as needed. If the file is swapped out prior to all sections being mapped, it could cause major instability.
I am looking for different ways to pause and resume programmatically a particular process via its process ID under Windows XP.
Process suspend/resume tool does it with SuspendThread / ResumeThread but warns about multi-threaded programs and deadlock problems.
PsSuspend looks okay, but I wonder if it does anything special about deadlocks or uses another method?
Prefered languages : C++ / Python
If you "debug the debugger" (for instance, using logger.exe to trace all API calls made by windbg.exe), it appears that the debugger uses SuspendThread()/ResumeThread() to suspend all of the threads in the process being debugged.
PsSuspend may use a different way of suspending processes (I'm not sure), but it is still possible to hang other processes: if the process you're suspending is holding a shared synchronization object that is needed by another process, you may block that other process from making any progress. If both programs are well-written, they should recover when you resume the one that you suspended, but not all programs are well-written. And if this causes your program that is doing the suspending to hang, then you have a deadlock.
I'm not sure if this does the job, but with ProcessExplorer from MS Systernals you can suspend a process.
It's been said here: https://superuser.com/a/155263 and I found it there too.
read here and you also have psutil for python that you can use it like that:
>>> import psutil
>>> pid = 7012
>>> p = psutil.Process(pid)
>>> p.suspend()
>>> p.resume()
I tested http://www.codeproject.com/KB/threads/pausep.aspx on few softwares:
it works fine.
PsSuspend and Pausep are two valid options.
So, after I found about UniversalPauseButton, Googling for this ("windows SIGSTOP"), getting this question as the first search result (thanks Ilia K. your comment did its job), and reading the answers, I went back to checkout the code.
Apparently, it uses undocumented NT kernel and Win32 APIs _NtSuspendProcess, _NtResumeProcess and _HungWindowFromGhostWindow.
PsSuspend, the utility you mentioned and linked to probably uses these APIs, I couldn't verify this, the source code isn't supplied, only executables and a EULA, you can probably figure that out by disassembling the binary but it's against the EULA.
so, to answer your specific question, checkout UniversalPauseButton's main.cpp, basically you call _NtSuspendProcess(ProcessHandle) and _NtResumeProcess(ProcessHandle), ProcessHandle being the handle of the process you want to pause or resume.
I think there is a good reason why there is no SuspendProcess() function in Windows. Having such a function opens the door for an unstable system. You shall not suspend a process unless you created that process yourself.
If you wrote that process yourself, you could use an event (see ::SetEvent() etc. in MSDN) or another kind of messaging to trigger a pause command in the process.