I have a set of tasks that I need to run from a Ruby script, however one particular task always waits for EOF on STDIN before quitting.
Obviously this causes the script to hang while waiting for the child process to end.
I have the process ID of the child process, but not a pipe or any kind of handle to it. How could I open a handle to the STDIN of a process to send EOF to it?
EDIT: Given that you aren't starting the script, a solution that occurs to me is to put $stdin under your control while using your gem. I suggest something like:
old_stdin = $stdin.dup
# note that old_stdin.fileno is non-0.
# create a file handle you can use to signal EOF
new_stdin = File::open('/dev/null', 'r')
# and make $stdin use it, instead.
$stdin.reopen(new_stdin)
new_stdin.close
# note that $stdin.fileno is still 0, though now it's using /dev/null for input.
# replace with the call that runs the external program
system('/bin/cat')
# "cat" will now exit. restore the old state.
$stdin.reopen(old_stdin)
old_stdin.close
If your ruby script is creating the tasks, it can use IO::popen. For example, cat, when run with no arguments, will wait for EOF on stdin before it exits, but you can run the following:
f = IO::popen('cat', 'w')
f.puts('hello')
# signals EOF to "cat"
f.close
Related
edit: I think I fixed the issue: https://gist.github.com/niuage/c0637b8dd10549a12b6a223dbd5f158a
I might have been missing the Process.wait, hence creating a lot of zombie processes.
I have a piece of code that's working most of the time, but "locks" itself after a while, probably because of a race condition.
My code
pipe = "goals.png"
(1..100).each do |time|
fork do
# This runs ffmpeg, and gets it to write 1 frame of a video to the pipe 'goals.png'
print "before ffmpeg"
`#{ffmpeg(time, score_crop_options, pipe)}`
exit
end
# Reads from 'pipe'
print "before read"
blob = File.read(pipe)
image = Rocket::Image.from_blob(blob)
# do stuff with image
end
Things to note:
#{ffmpeg(time, pipe)} writes to pipe, and is blocking until something reads from pipe
File.read(pipe) is blocking until something writes to pipe
My issue
edit: when the script is locked, and I try to read the pipe from another script, I get zsh: fork failed: resource temporarily unavailable. That's probably a good clue...
Most of the time, File.read(pipe) gets executed before the code in fork, so it works great, but after a little while the script just stops: it prints "before ffmpeg" and never gets to "before read"...
First, should I use threads instead of fork? And can I control the order the 2 statements (read and write) get run, to avoid a race condition? Or maybe it's not even about the race condition and I'm missing something?
The issue wasn't caused by a race condition, but too many zombie processes, since I wasn't calling Process.wait
The parent process should use Process.wait to collect the termination statuses of its children or use Process.detach to register disinterest in their status; otherwise, the operating system may accumulate zombie processes.
That's why I was getting zsh: fork failed: resource temporarily unavailable when trying to read from the pipe from another script probably.
Here's something that works:
(1..100) do
if fork
# Parent
image = read_image(pipe)
# do stuff with image
Process.wait # I think that's what was missing previously
else
# Child
Open3.popen3(command(time, score_crop_options, pipe)) do |stdin, stdout, stderr, wait_thr|
# stuff
end
exit!(0)
end
end
In Ruby, is it possible to prevent the standard input of a spawned child process from being attached to the terminal without having to capture the STDOUT or STDERR of that same process?
Backticks and x-strings (`...`, %x{...}) don't work because they capture STDIN.
Kernel#system doesn't work because it leaves STDIN attached to the
terminal (which intercepts signals like ^C and prevents them from
reaching my program, which is what I'm trying to avoid).
Open3 doesn't work because its methods capture either STDOUT or
both STDOUT and STDERR.
So what should I use?
If you’re on a platform that supports it, you could do this with pipe, fork and exec:
# create a pipe
read_io, write_io = IO.pipe
child = fork do
# in child
# close the write end of the pipe
write_io.close
# change our stdin to be the read end of the pipe
STDIN.reopen(read_io)
# exec the desired command which will keep the stdin just set
exec 'the_child_process_command'
end
# in parent
# close read end of pipe
read_io.close
# write what we want to the pipe, it will be sent to childs stdin
write_io.write "this will go to child processes stdin"
write_io.close
Process.wait child
I have a bash shell script. It writes out to a text file. Most of the it works find if I stop the script with a control-c at the command level. Sometimes the file that's been written to such as
echo "hello world" >myfile.txt
will end up being empty. So it it possible that when I hit control-c to stop the shell script running it is caught it at the instance where it's opening a write to the file and before it puts anything in it, it doesn't get the chance and leaves it empty?
If that's the case. What can I do in the bash shell script so that it will exit gracefully after it's written to the file and before it gets a chance to write to the file again, because it's doing this in a while loop. Thanks!
Yes, it's possible that you end up with an empty file.
A solution would be to trap the signal that's caused by ^C (SIGINT), and set a flag which you can check in your loop:
triggered=0
trap "triggered=1" SIGINT
while true
do
if [ $triggered = 1 ]
then
echo "quitting"
exit
fi
...do stuff...
done
EDIT: didn't realize that even though the shell's own SIGINT handling will get trapped, it will still pass the SIGINT to its subprocesses, and they'll get killed if they don't handle SIGINT themselves.
Since echo is a shell builtin, it might survive the killing, but I'm not entirely sure. A quick test seems to work okay (file is always written, whereas without trapping SIGINT, I occasionally end up with an empty file as well).
As #spbnick suggests in the comments, on Linux you can use the setsid command to create a new process group for any subprocesses you start, which will prevent them from being killed by a SIGINT sent to the shell.
In Ruby, I'm running a system("command here") that is constantly watching changes for files, similar to tail. I'd like my program to continue to run and not halt at the system() call. Is there a way in Ruby to create another process so both can run independently, output results to the terminal, and then when you exit the program all processes the application created are removed?
Just combine spawn and waitall:
spawn 'sleep 6'
spawn 'sleep 8'
Process.waitall
You don't want to use system as that waits for the process to complete. You could use spawn instead and then wait for the processes (to avoid zombies). Then, when you want to exit, send a SIGTERM to your spawned processes. You could also use fork to launch your child processes but spawn is probably easier if you're using external programs.
You could also use process groups instead of tracking all the process IDs, then a single Process.kill('TERM', -process_group_id) call would take care of things. Your child processes should end up in the same process group but there is Process.setpgid if you need it.
Here's an example that uses fork (easier to get it all wrapped in one package that way).
def launch(id, sleep_for)
pid = Process.fork do
while(true)
puts "#{id}, pgid = #{Process.getpgid(Process.pid())}, pid = #{Process.pid()}"
sleep(sleep_for)
end
end
# No zombie processes please.
Process.wait(pid, Process::WNOHANG)
pid
end
# These just forward the signals to the whole process group and
# then immediately exit.
pgid = Process.getpgid(Process.pid())
Signal.trap('TERM') { Process.kill('TERM', -pgid); exit }
Signal.trap('INT' ) { Process.kill('INT', -pgid); exit }
launch('a', 5)
launch('b', 3)
while(true)
puts "p, pgid = #{Process.getpgid(Process.pid())}, pid = #{Process.pid()}"
sleep 2
end
If you run that in one terminal and then kill it from another (using the shell's kill command)you'll see that the children are also killed. If you remove the "forward this signal to the whole process group" Signal.trap stuff, then a simple SIGTERM will leave the children still running.
All of this assumes that you're working on some sort of Unixy system (such as Linux or OSX), YMMV anywhere else.
One more vote for using Spawn. We use it in Production a lot and it's very stable.
I would like to write a Ruby script that runs a daemon Ruby process, so that I can do something like the following:
$ task start
Started...
# start a daemon to do useful work; don't block
# Some time later:
$ task end
Finished.
What's the best way to go about this?
Use the Process.daemon method when you want your process to detach from the terminal and stop being able to send output. To end the process, you will have to send it a signal. Most programs handle this by using a file that contains the PID.