I'm refactoring a bit of concurrent processing in my Ruby on Rails server (running on Linux) to use Spawn. Spawn::fork_it documentation claims that forked processes can still be waited on after being detached: https://github.com/tra/spawn/blob/master/lib/spawn.rb (line 186):
# detach from child process (parent may still wait for detached process if they wish)
Process.detach(child)
However, the Ruby Process::detach documentation says you should not do this: http://www.ruby-doc.org/core/classes/Process.html
Some operating systems retain the status of terminated child processes until the parent collects that status (normally using some variant of wait(). If the parent never collects this status, the child stays around as a zombie process. Process::detach prevents this by setting up a separate Ruby thread whose sole job is to reap the status of the process pid when it terminates. Use detach only when you do not intent to explicitly wait for the child to terminate.
Yet Spawn::wait effectively allows you to do just that by wrapping Process::wait. On a side note, I specifically want to use the Process::waitpid2 method to wait on the child processes, instead of using the Spawn::wait method.
Will detach-and-wait not work correctly on Linux? I'm concerned that this may cause a race condition between the detached reaper thread and the waiting parent process, as to who collects the child status first.
The answer to this question is there in the documentation. Are you writing code for your own use in a controlled environment? Or to be used widely by third parties? Ruby is written to be widely used by third parties, so their recommendation is to not do something that could fail on "some operating systems". Perhaps the Spawn library is designed primarily for use on Linux machines and tested only on a small subset thereof where this tactic works.
If you're distributing the code you're writing to be used by anyone and everyone, I would take Ruby's approach.
If you control the environment where this code will be run, I would write two tests:
A test that spawns a process, detaches it and then waits for it.
A test that spawns a process and then just waits for it.
Count the failure rate for both and if they are equal (within a margin that you feel is acceptable), go for it!
Related
I am working on a program to start a long running process (afplay, with long sound files) using system() and at a later time possibly decide to terminate this process. It seems that it would be straightforward to invoke a system("prog") call and then later a system("killall prog") call. Using pthreads, I fire up a thread to invoke the initial system("prog") call and then later if application detects that its time to terminate early, the main thread will call system("killall prog"). Through print statements I can see that the main thread properly detects the logic to stop but the subsequent system call blocks until the original system call is finished (the main thread doesn't appear to block until this time, other activity does progress past the thread creation for the initial system call). If I try the killall from a separate shell after my program invokes the prog, killlall works (as you'd expect). I know that macOS has requirements on programs that interact with the ui libraries need to handle such activity from the main thread only. Are there other requirements for programs shelling out to system(3) that I clearly am ignorant of?
On windows, the only difference in the code is the choices for "prog", and the behavior works as I expect.
system() is expected to block until the launched program exits -- if it didn't, there would be no way for system() to return the exit-status of the child process as part of its return value.
If you want your thread to continue executing in parallel with the child process, you will need to use a different API (typically fork() followed by calling exec() from the child-process's branch of the fork) instead.
I am looking for a cross platform (various flavours of Unix, including Linux) to find and kill all processes spawned by my program. For Linux, I can walk to /proc to obtain this information, and I am sure I can find somethinf similar for OS X and *BSD. But I'd prefer if there were a standard library for this.
Background: I am writing a custom job schedular which needs to terminate the jobs that don't complete within a given period of time. Simply killing (SIGTERM, followed by SIGKILL- if the formar is not ignored) the child process works fine when the job doesn't spawn any other process or handles SIGTERM properly and takes care of the cleanup. But I don't control the jobs - and I know at least some that are poorly written. In the latter case, the system is left with a bunch of orphaned process which keep holdin on to certain resouces and cause all sorts of problems.
Any pointer to libraries or some cross platform way of doing this would be welcome.
I have this little test program that tracks PID's as they are created and shut down.
I am investigating a problem that my program has found and would like to ask you about this
in order to have a better idea on what's going on.
When a windows process is started, it gets a PID but when the process is shut down, does the PID
become retired (like a star basketballer's jersey number) or is it possible for a new, entirely
unrelated, process to be created under that released PID?
Thanks
Yes, process IDs may be recycled by the system. They become available for this as soon as the last handle to the process has been closed.
Raymond Chen discussed this matter here: When does a process ID become available for reuse?
The process ID is a value associated with the process object, and as
long as the process object is still around, so too will its process
ID. The process object remains as long as the process is still running
(the process implicitly retains a reference to itself) or as long as
somebody still has a handle to the process object.
If you think about it, this makes sense, because as long as there is
still a handle to the process, somebody can call WaitForSingleObject
to wait for the process to exit, or they can call GetExitCodeProcess
to retrieve the exit code, and that exit code has to be stored
somewhere for later retrieval.
When all handles are closed, then the kernel knows that nobody is
going to ask whether the process is still running or what its exit
code is (because you need a handle to ask those questions). At which
point the process object can be destroyed, which in turn destroys the
process ID.
I ran a test for about an hour and in that time 302 processes exits and 70 of them had PIDs in common (same PID was used for a new process). So that would say they are reused frequently.
Evidently, if the process is terminated, its PID is available for reuse.
http://msdn.microsoft.com/en-us/library/windows/desktop/ms683215%28v=vs.85%29.aspx
Remarks
Until a process terminates, its process identifier uniquely identifies it on the system. For more information about access rights, see Process Security and Access Rights.
I want to increase the throughput of a script which does net I/O (a scraper). Instead of making it multithreaded in ruby (I use the default 1.9.1 interpreter), I want to launch multiple processes. So, is there a system for doing this to where I can track when one finishes to re-launch it again so that I have X number running at any time. ALso some will run with different command args. I was thinking of writing a bash script but it sounds like a potentially bad idea if there already exists a method for doing something like this on linux.
I would recommend not forking but instead that you use EventMachine (and the excellent em-http-request if you're doing HTTP). Managing multiple processes can be a bit of a handful, even more so than handling multiple threads, but going down the evented path is, in comparison, much simpler. Since you want to do mostly network IO, which consist mostly of waiting, I think that an evented approach would scale as well, or better than forking or threading. And most importantly: it will require much less code, and it will be more readable.
Even if you decide on running separate processes for each task, EventMachine can help you write the code that manages the subprocesses using, for example, EventMachine.popen.
And finally, if you want to do it without EventMachine, read the docs for IO.popen, Open3.popen and Open4.popen. All do more or less the same thing but give you access to the stdin, stdout, stderr (Open3, Open4), and pid (Open4) of the subprocess.
You can try fork http://ruby-doc.org/core/classes/Process.html#M003148
You can get the PID in return and see if this process run again or not.
If you want manage IO concurrency. I suggest you to use EventMachine.
You can either
implement (or find an equivalent gem) a ThreadPool (ProcessPool, in your case), or
prepare an array of all, let's say 1000 tasks to be processed, split it into, say 10 chunks of 100 tasks (10 being the number of parallel processes you want to launch), and launch 10 processes, of which each process right away receives 100 tasks to process. That way you don't need to launch 1000 processes and control that not more than 10 of them work at the same time.
I would like to run arbitrary console-based sub-processes and manage them from a single master process. The console based sub-processes communicate via stdin, stdout and stderr, and if you run them in a genuine console they terminate cleanly when you press CTRL+C. Some of them may in fact be a tree of processes, such as a batch script that runs an executable which may in turn run another executable to do some work. I would like to redirect their standard I/O (for example, so that I can show their output in a GUI window) and in certain circumstances to send them a CTRL+C event so that they will give up and terminate cleanly.
The following two diagrams show first the normal structure - one master process has four worker sub-processes, and some of those workers have their own subprocesses; and then what should happen when one of the workers needs to be stopped - it and all of its children should get the CTRL+C event, but no other processes should receive the CTRL+C event.
(source: livejournal.com)
Additionally, I would much prefer that there are no extra windows visible to the user.
Here's what I've tried (note that I'm working in Python, but solutions for C would still be helpful):
Spawning an extra intermediate process with CREATE_NEW_CONSOLE, and then having it spawn the worker process. Then have it call GenerateConsoleCtrlEvent(CTRL_C_EVENT, 0) when we want to kill the worker. Unfortunately, CREATE_NEW_CONSOLE seems to prevent me from redirecting the standard I/O channels, so I'm left with no easy way to get the output back to the main program.
Spawning an extra intermediate process with CREATE_NEW_PROCESS_GROUP, and then having it spawn the worker process. Then have it call GenerateConsoleCtrlEvent(CTRL_C_EVENT, 0) when we want to kill the worker. Somehow, this manages to send the CTRL+C only to the master process, which is completely useless. On closer inspection, GenerateConsoleCtrlEvent says that CTRL+C cannot be sent to process groups.
Spawning the subprocess with CREATE_NEW_PROCESS_GROUP. Then call GenerateConsoleCtrlEvent(CTRL_BREAK_EVENT, pid) to kill the worker. This is not ideal, because CTRL+BREAK is less friendly than CTRL+C and will probably result in a messier termination. (E.g. if it's a Python process, no KeyboardInterrupt can be caught and no finally blocks run.)
Is there any good way to do what I want? I can see that I could theoretically build on the first attempt and find some other way to communicate between the processes, but I am worried it will turn out to be extremely awkward. Are there good examples of other programs that achieve the same effect? It seems so simple that it can't be all that uncommon a requirement.
I don't know about managing/redirecting stdin et. al., but for managing the subprocess tree
have you considered using the Windows Job Objects api?
There are several other questions about managing process trees (How do I automatically destroy child processes in Windows? Performing equivalent of “Kill Process Tree” in c++ on windows) and it looks like the cleanest method if you can use it.
Chapter 5 of Windows Via C/C++ by Jeffery Richter has a good discussion on using CreateJobObject and the related APIs.