I found in Sidekiq the next piece of code related to process demonization:
files_to_reopen = []
ObjectSpace.each_object(File) do |file|
files_to_reopen << file unless file.closed?
end
::Process.daemon(true, true)
files_to_reopen.each do |file|
begin
file.reopen file.path, "a+"
file.sync = true
rescue ::Exception
end
end
[$stdout, $stderr].each do |io|
File.open(options[:logfile], 'ab') do |f|
io.reopen(f)
end
io.sync = true
end
$stdin.reopen('/dev/null')
https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/cli.rb#L191-L212
I can't understand why do we have to reopen files as far as we inherit file descriptors on double fork? Does some specific case exist to do this? Authors of the "Advanced Programming in the UNIX Environment" book, chapter "Daemon Processes", say only about closing the unneeded file descriptors:
Unneeded file descriptors should be closed. This prevents the daemon from holding open any descriptors that it may have inherited from its parent (which could be a shell or some other process). We can use our open_max function (Figure 2.17) or the getrlimit function (Section 7.11) to determine the highest descriptor and close all descriptors up to that value.
I believe much of that code was inspired by Unicorn. Eric Wong, author of Unicorn, is Mr. Linux in the Ruby world and usually knows all the tricks to do daemonization correctly.
Of course, in the broader picture, don't daemonize. You should start Sidekiq with a proper process supervisor: systemd, upstart, runit, foreman, etc.
Related
I have a basic understanding of popen before now but it seems it has changed completely.
Please refer the example to know why?
# process.rb
IO.popen("ruby test_ex.rb","w") do |io|
io.write("#{Process.pid} hello")
io.close_write
## this does not work.
##io.readlines
end
## text_ex.rb
def readWrite
#string = gets()
puts "#{Process.pid} -- #{#string}"
end
readWrite
Now I understand in write mode the STDOUT(of popen.rb) will be writable end of the pipe and STDIN (of text_ex.rb) will be the readable end of the pipe.
All is good here.
But let see the other example
my_text = IO.popen("ssh user#host 'bash'", "w+")
my_text.write("hostname")
my_text.close_write
my_rtn = my_text.readlines.join('\n')
my_text.close
puts my_rtn
Ok, now what is different over here?
The popen start a child process(i.e ssh) send the hostname.
Now, I fail to understand how does the STDOUT of the child process(i.e ssh) is available to the parent process i.e how does the readlines work over here and does not work in my earlier example.
Thanks
The difference is in the second argument to popen: "w" versus "w+". You can read more here in the docs:
"w" Write-only, truncates existing file
to zero length or creates a new file for writing.
"w+" Read-write, truncates existing file to zero length
or creates a new file for reading and writing.
The notion of "truncating" doesn't really apply to pipes, but the fact that you need read-write mode does.
I am writing to a file instance. While the program is still running, the file is always empty. When I check the file after the script has executed, the file has content.
class A
def initialize
#file_ref=File.new("/user/shared/ruby/ruby-example/test.html","w+")
end
def fill
#file_ref.write("whatever\nwhatever\nwhatever\n")
end
end
The Main script:
require_relative 'A'
a=A.new
a.fill
puts File.size("/user/shared/ruby/ruby-example/test.html")
After the A instance has done its job, the puts statement will print "0" as if the file is empty. Indeed it is during program execution, but if I start irb:
puts File.size("/user/shared/ruby/ruby-example/test.html")
# => 27
$ cat test.html
whatever
whatever
whatever
Is my code wrong?
Is it normal that streams are flushed only after the execution of a process?
Ruby flushes IO buffers when you call IO#close or IO#flush. Since you are not calling neither close nor flush the buffers are flushed when the program terminates and the opened file descriptors are released.
Given your simple example a possible solution is:
class A
def initialize
#file_ref_name = '/user/shared/ruby/ruby-example/test.html'
end
def fill
File.open(#file_ref_name, 'w+') do |file|
file.write("whatever\nwhatever\nwhatever\n")
end
end
end
Passing a block to IO#open makes the opened file (the file variable in this example) to be closed (and therefore flushed) once the execution of the block terminates.
Please note that Ruby (to my knowledge since version 1.9) features a one liner shortcut for simple file writes as well, flush included:
File.write('/path/to/file.txt', 'content')
i'm using ruby 1.9.2p180 (2011-02-18 revision 30909)
in Order to do the logging i use the logging gem.
My program has two blocks, which are used as daemons.
But logging from these blocks results in an error and nothing is written to the logfile:
log shifting failed. closed stream
log writing failed. closed stream
Here is what happens in the code:
log = Logger.new(logbase + 'logfile.log', 'monthly')
log.level = Logger::INFO
proc = Daemons.call(options) do
# [...]
log.info "Any Logmessage"
# [...]
end
Any Idea, whats wrong there?
The Daemons gem closes all file descriptors when it daemonizes the process. So any logfiles that were opened before the Daemons block will be closed inside the forked process.
And since you can't write to closed file descriptors -> errors.
You can read more about what happens when you daemonize a process by reading the chapter:
What does daemons internally do with my daemons?
http://daemons.rubyforge.org/Daemons.html
The solution is to open the logfile inside the daemon block instead of outside of it. That should fix it. But note that daemonizing changes the working directory to /, so take that into account when referencing logfile paths.
A solution which successfully works in delayed_job gem includes extracting all open files before fork, and reopening them later.
An adjusted extract from delayed_job:
#files_to_reopen = []
ObjectSpace.each_object(File) do |file|
#files_to_reopen << file unless file.closed?
end
Daemons.run_proc('some_process') do
#files_to_reopen.each do |file|
file.reopen file.path, 'a+'
file.sync = true
end
# Your code
end
Just some food for thought about how necessary it is to close the files that I opened explicitly in the code. I came from a background of programming in C and C++, and starting to navigate my way through Ruby. Thanks in advance for your feedback.
from_file, to_file = ARGV
script = $0
puts "Copying from #{from_file} to #{to_file}"
File.open(to_file, 'w').write(File.open(from_file).read())
puts "Alright, all done."
Not closing files is always bad practice unless you are using something like the with statement in python.
While a scripting language will usually close open files on exit, it's cleaner to do it as soon as you are done with the file - especially when writing to it.
Apparently Ruby has something similar to python's with:
File.open(from_file, 'r') do |f_in|
File.open(to_file, 'w') do |f_out|
f_out.write(f_in.read)
end
end
Relevant docs: http://ruby-doc.org/core-1.9.3/File.html#method-c-open
Here's a shorter version:
File.write to_file, File.read(from_file)
This code (Matheus Moreira) closes files automatically:
File.write to_file, File.read(from_file)
There are no ways to close files in this code:
File.open(to_file, 'w').write(File.open(from_file).read())
I guess automatically closing too.
It's a good answer but it's more 'ruby' to put the output file on the outer block and use << :
File.open(to_file, 'w') do |f_out|
f_out << File.open(from_file){|f| f.read}
end
note how you do not need the 'r' when reading.
I want to write a Ruby program that will always be running in the background (a daemon) on my Mac.
Can someone point me in the right direction on how this would be done?
Ruby 1.9.x has now the following:
Process.daemon
Put it in your code and that's it.
Taken from "Daemon Processes in Ruby."
Use Daemonize.rb
require 'daemons'
Daemons.daemonize
Very simple sample: http://github.com/utkarsh2012/backitup/blob/master/backitup.rb
How to install daemons gem:
gem install daemons
Ah, Google to the rescue! Check out
http://fitzgeraldsteele.wordpress.com/2009/05/04/launchd-example-start-web-server-at-boot-time/
wherein a helpful blogger provides an example of writing a launchd plist to launch a ruby Web application server.
This is a module to daemonize your code. Here's an offshoot that wraps an existing script.
Essentially it boils down to this (from Travis Whitton's Daemonize.rb, the first link above, modified for some program I wrote ages ago):
private
# This method causes the current running process to become a daemon
# If closefd is true, all existing file descriptors are closed
def daemonize(pathStdErr, oldmode=0, closefd=false)
srand # Split rand streams between spawning and daemonized process
safefork and exit# Fork and exit from the parent
# Detach from the controlling terminal
unless sess_id = Process.setsid
raise 'Cannot detach from controlled terminal'
end
# Prevent the possibility of acquiring a controlling terminal
if oldmode.zero?
trap 'SIGHUP', 'IGNORE'
exit if pid = safefork
end
Dir.chdir "/" # Release old working directory
File.umask 0000 # Insure sensible umask
if closefd
# Make sure all file descriptors are closed
ObjectSpace.each_object(IO) do |io|
unless [STDIN, STDOUT, STDERR].include?(io)
io.close rescue nil
end
end
end
STDIN.reopen "/dev/null" # Free file descriptors and
STDOUT.reopen "/dev/null" # point them somewhere sensible
STDERR.reopen pathStdErr, "w" # STDOUT/STDERR should go to a logfile
return oldmode ? sess_id : 0 # Return value is mostly irrelevant
end
# Try to fork if at all possible retrying every 5 sec if the
# maximum process limit for the system has been reached
def safefork
tryagain = true
while tryagain
tryagain = false
begin
if pid = fork
return pid
end
rescue Errno::EWOULDBLOCK
sleep 5
tryagain = true
end
end
end
Need to see the daemons-rails gem for Rails 3 (based on rails_generator):
https://github.com/mirasrael/daemons-rails
Possible to generate daemon stub like this:
rails generate daemon <name>
Features:
individual control script per daemon
rake:daemon command per daemon
capistrano friendly
app-wide control script
monitoring API
possible multiple daemon sets