I have a pieces of code where i spawn off children processes to make it more efficient. However, they appear to create all sorts of zombie processes which block sockets and bring down the site.
spawn(:method => :thread) do
if #login_user.suggested_group_info.new_record?
xxx
end
end
1) Why is this creating zombie processes?
2) How could i write the code such that i make sure i kill the process before it becomes a zombie?
You have to save the PID of the spawned process and execute the waitpid(2) system call upon it after it dies. (I don't know how Ruby does this.)
You can also trap for the child shutdown which will clean up the zombie process
trap("CLD") {
pid = Process.wait
puts "Child pid #{pid}: terminated"
}
Related
The following code forks the main processes and runs a command in backticks. The kill at the end of the script only kills the forked process but not it's child processes (i.e. the sleep command).
pid = fork do
Thread.new do
`sleep 20`
end
end
sleep(1)
Process.kill("HUP",pid)
Is there a way to kill all child processes (generated by backtick commands in threads in the forked process) other than searching through the process tree?
Behind the scene both system and backtick operations use fork to fork
the current process and then they execute the given operation using
exec .
Since exec replaces the current process it does not return anything if
the operation is a success. If the operation fails then
`SystemCallError is raised.
http://blog.bigbinary.com/2012/10/18/backtick-system-exec-in-ruby.html
You can use
pid = Process.spawn('sleep 20')
to get the PID of the process immediately. Your code above would change to:
pid = Process.spawn('sleep 20')
sleep(1)
Process.kill('HUP',pid)
I am currently reading up on some more details on Bash scripting and especially process management here. In the section on "PIDs and Parents" I found the following statement:
A process's PID will NEVER be freed up for use after the process dies UNTIL the parent process waits for the PID to see whether it ended and retrieve its exit code.
So if I understand this correctly, if I start an process in a bash script, then the process terminates, that the PID cannot be used by any other process. Wouldn't this mean, that if I have a long running script, which repeatedly starts other sub-processes but never waits on them, that I'll eventually have a resource leak, because the used PIDs will not be returned back to the system?
How about if I actually wait for the other process, but the wait get's cancelled by a trap. Would this wait somehow still free up the PID, or do I have to wait again after the trap has been caught?
Luckily you won't. I can't tell you exactly why but you can easily test this. Run the following script (stop with Ctrl+C):
#!/bin/bash
while true; do
sleep 5 &
sleep 1
done
You can see you get no zombies (leaked PIDs) after 6+ seconds. To see some zombies use the following python code (again, stop with Ctrl+C):
#!/usr/bin/python
import subprocess, time
pl = []
while True:
pl.append(subprocess.Popen(["sleep", "5"]))
time.sleep(1)
After 6 seconds you'll see one zombie:
ps xaw | grep 'sleep'
...
26470 pts/2 Z+ 0:00 [sleep] <defunct>
...
My guess is that bash does wait and stores the results reaping the zombile processes with or without the builtin wait command. For the python script, if you remove the pl.append part the garbage collection releases the objects and does it's magic again reaping the zombies. Just for info a child may never become a zombie (from wikipedia, Zombie process):
...if the parent explicitly ignores SIGCHLD by setting its handler to SIG_IGN (rather
than simply ignoring the signal by default) or has the SA_NOCLDWAIT flag set, all
child exit status information will be discarded and no zombie processes will be left.
You don't have to explicitly wait on foreground processes because the shell in which your script is running waits on them. The next process won't start until the previous one finishes.
If you start many long running background processes, you could use all available PIDs, but that's subject to the limit of ulimit -u (which could be unlimited).
In Ruby, I'm running a system("command here") that is constantly watching changes for files, similar to tail. I'd like my program to continue to run and not halt at the system() call. Is there a way in Ruby to create another process so both can run independently, output results to the terminal, and then when you exit the program all processes the application created are removed?
Just combine spawn and waitall:
spawn 'sleep 6'
spawn 'sleep 8'
Process.waitall
You don't want to use system as that waits for the process to complete. You could use spawn instead and then wait for the processes (to avoid zombies). Then, when you want to exit, send a SIGTERM to your spawned processes. You could also use fork to launch your child processes but spawn is probably easier if you're using external programs.
You could also use process groups instead of tracking all the process IDs, then a single Process.kill('TERM', -process_group_id) call would take care of things. Your child processes should end up in the same process group but there is Process.setpgid if you need it.
Here's an example that uses fork (easier to get it all wrapped in one package that way).
def launch(id, sleep_for)
pid = Process.fork do
while(true)
puts "#{id}, pgid = #{Process.getpgid(Process.pid())}, pid = #{Process.pid()}"
sleep(sleep_for)
end
end
# No zombie processes please.
Process.wait(pid, Process::WNOHANG)
pid
end
# These just forward the signals to the whole process group and
# then immediately exit.
pgid = Process.getpgid(Process.pid())
Signal.trap('TERM') { Process.kill('TERM', -pgid); exit }
Signal.trap('INT' ) { Process.kill('INT', -pgid); exit }
launch('a', 5)
launch('b', 3)
while(true)
puts "p, pgid = #{Process.getpgid(Process.pid())}, pid = #{Process.pid()}"
sleep 2
end
If you run that in one terminal and then kill it from another (using the shell's kill command)you'll see that the children are also killed. If you remove the "forward this signal to the whole process group" Signal.trap stuff, then a simple SIGTERM will leave the children still running.
All of this assumes that you're working on some sort of Unixy system (such as Linux or OSX), YMMV anywhere else.
One more vote for using Spawn. We use it in Production a lot and it's very stable.
Given a very simple ruby script:
child = fork do
system 'sleep 10000'
end
5.times do
sleep 1
puts "send kill to #{child}"
Process.kill("QUIT", child)
end
QUIT signal is just lost. Where does it go? Something with default handler which just ignores it?
How to send signal to all processes created by that fork? Is it possible to do that without searching for all child processes?
The problem is that the system call creates yet another child process running the given command in a subshell, so there are actually three processes running in your example. Additionally, the Ruby Kernel#system command is implemented via the standard C function system(3), which calls fork and exec to create the new process and (on most systems) ignores SIGINT and SIGQUIT, and blocks SIGCHLD.
If you simply call sleep(10000) instead of system("sleep 10000") then things should work as you expect. You can also trap SIGQUIT in the child to handle it gracefully:
child = fork do
Signal.trap("QUIT") { puts "CHILD: ok, quitting time!"; exit }
sleep(10000)
end
If you really need to use a "system" call from the child process then you might be better off using an explicit fork/exec pair (instead of the implicit ones in the system call), so that you can perform your own signal handling in the third forked child.
I think that you are sending signal to fork process corectly. I think that the problem is with the system command. System command creates new fork and waits until it ends and I think that this waiting is blocking your quit signal. If you run your example as test.rb you'll see three processes:
test.rb
test.rb
sleep 10000
If you send signal "TERM" or "KILL" instead of "QUIT" the second test.rb will die but sleep 10000 will continue!
i'm trying to set up a server for integration tests (specs actually) via ruby and can't figure out how to control the process.
so, what i'm trying to do is:
run a rake task for my gem that executes the integration specs
the task needs to first start a server (i use webrick) and then run the specs
after executing the specs it should kill the webrick so i'm not left with some unused background process
webrick is not a requirement, but it's included in the ruby standard library so being able to use it would be great.
hope anyone is able to help!
ps. i'm running on linux, so having this work for windows is not my main priority (right now).
The standard way is to use the system functions fork (to duplicate the current process), exec (to replace the current process by an executable file), and kill (to send a signal to a process to terminate it).
For example :
pid = fork do
# this code is run in the child process
# you can do anything here, like changing current directory or reopening STDOUT
exec "/path/to/executable"
end
# this code is run in the parent process
# do your stuffs
# kill it (other signals than TERM may be used, depending on the program you want
# to kill. The signal KILL will always work but the process won't be allowed
# to cleanup anything)
Process.kill "TERM", pid
# you have to wait for its termination, otherwise it will become a zombie process
# (or you can use Process.detach)
Process.wait pid
This should work on any Unix like system. Windows creates process in a different way.
I just had to do something similar and this is what I came up with. #Michael Witrant's answer got me started, but I changed some things like using Process.spawn instead of fork (newer and better).
# start spawns a process and returns the pid of the process
def start(exe)
puts "Starting #{exe}"
pid = spawn(exe)
# need to detach to avoid daemon processes: http://www.ruby-doc.org/core-2.1.3/Process.html#method-c-detach
Process.detach(pid)
return pid
end
# This will kill off all the programs we started
def killall(pids)
pids.each do |pid|
puts "Killing #{pid}"
# kill it (other signals than TERM may be used, depending on the program you want
# to kill. The signal KILL will always work but the process won't be allowed
# to cleanup anything)
begin
Process.kill "TERM", pid
# you have to wait for its termination, otherwise it will become a zombie process
# (or you can use Process.detach)
Process.wait pid
rescue => ex
puts "ERROR: Couldn't kill #{pid}. #{ex.class}=#{ex.message}"
end
end
end
# Now we can start processes and keep the pids for killing them later
pids = []
pids << start('./someprogram')
# Do whatever you want here, run your tests, etc.
# When you're done, be sure to kill of the processes you spawned
killall(pids)
That's about all she wrote, give it a try and let me know how it works.
I have tried fork, but it has kind of problems when ActiveRecord is involved in both the processes. I would suggest Spawn plugin (http://github.com/tra/spawn). It does fork only but takes care of ActiveRecord.