i'm using ruby 1.9.2p180 (2011-02-18 revision 30909)
in Order to do the logging i use the logging gem.
My program has two blocks, which are used as daemons.
But logging from these blocks results in an error and nothing is written to the logfile:
log shifting failed. closed stream
log writing failed. closed stream
Here is what happens in the code:
log = Logger.new(logbase + 'logfile.log', 'monthly')
log.level = Logger::INFO
proc = Daemons.call(options) do
# [...]
log.info "Any Logmessage"
# [...]
end
Any Idea, whats wrong there?
The Daemons gem closes all file descriptors when it daemonizes the process. So any logfiles that were opened before the Daemons block will be closed inside the forked process.
And since you can't write to closed file descriptors -> errors.
You can read more about what happens when you daemonize a process by reading the chapter:
What does daemons internally do with my daemons?
http://daemons.rubyforge.org/Daemons.html
The solution is to open the logfile inside the daemon block instead of outside of it. That should fix it. But note that daemonizing changes the working directory to /, so take that into account when referencing logfile paths.
A solution which successfully works in delayed_job gem includes extracting all open files before fork, and reopening them later.
An adjusted extract from delayed_job:
#files_to_reopen = []
ObjectSpace.each_object(File) do |file|
#files_to_reopen << file unless file.closed?
end
Daemons.run_proc('some_process') do
#files_to_reopen.each do |file|
file.reopen file.path, 'a+'
file.sync = true
end
# Your code
end
Related
I found in Sidekiq the next piece of code related to process demonization:
files_to_reopen = []
ObjectSpace.each_object(File) do |file|
files_to_reopen << file unless file.closed?
end
::Process.daemon(true, true)
files_to_reopen.each do |file|
begin
file.reopen file.path, "a+"
file.sync = true
rescue ::Exception
end
end
[$stdout, $stderr].each do |io|
File.open(options[:logfile], 'ab') do |f|
io.reopen(f)
end
io.sync = true
end
$stdin.reopen('/dev/null')
https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/cli.rb#L191-L212
I can't understand why do we have to reopen files as far as we inherit file descriptors on double fork? Does some specific case exist to do this? Authors of the "Advanced Programming in the UNIX Environment" book, chapter "Daemon Processes", say only about closing the unneeded file descriptors:
Unneeded file descriptors should be closed. This prevents the daemon from holding open any descriptors that it may have inherited from its parent (which could be a shell or some other process). We can use our open_max function (Figure 2.17) or the getrlimit function (Section 7.11) to determine the highest descriptor and close all descriptors up to that value.
I believe much of that code was inspired by Unicorn. Eric Wong, author of Unicorn, is Mr. Linux in the Ruby world and usually knows all the tricks to do daemonization correctly.
Of course, in the broader picture, don't daemonize. You should start Sidekiq with a proper process supervisor: systemd, upstart, runit, foreman, etc.
I am writing to a file instance. While the program is still running, the file is always empty. When I check the file after the script has executed, the file has content.
class A
def initialize
#file_ref=File.new("/user/shared/ruby/ruby-example/test.html","w+")
end
def fill
#file_ref.write("whatever\nwhatever\nwhatever\n")
end
end
The Main script:
require_relative 'A'
a=A.new
a.fill
puts File.size("/user/shared/ruby/ruby-example/test.html")
After the A instance has done its job, the puts statement will print "0" as if the file is empty. Indeed it is during program execution, but if I start irb:
puts File.size("/user/shared/ruby/ruby-example/test.html")
# => 27
$ cat test.html
whatever
whatever
whatever
Is my code wrong?
Is it normal that streams are flushed only after the execution of a process?
Ruby flushes IO buffers when you call IO#close or IO#flush. Since you are not calling neither close nor flush the buffers are flushed when the program terminates and the opened file descriptors are released.
Given your simple example a possible solution is:
class A
def initialize
#file_ref_name = '/user/shared/ruby/ruby-example/test.html'
end
def fill
File.open(#file_ref_name, 'w+') do |file|
file.write("whatever\nwhatever\nwhatever\n")
end
end
end
Passing a block to IO#open makes the opened file (the file variable in this example) to be closed (and therefore flushed) once the execution of the block terminates.
Please note that Ruby (to my knowledge since version 1.9) features a one liner shortcut for simple file writes as well, flush included:
File.write('/path/to/file.txt', 'content')
Out of curiosity I tried file-tail ruby library to see how it works with ruby code. But, the code doesn't seem to be working.
Here is what I tired(logger.rb):
$:.unshift File.dirname(__FILE__)
filename = 'logger.log'
require "file-tail"
File.open(filename) do |log|
log.extend(File::Tail)
log.interval = 10
log.backward(10)
log.tail { |line| puts line }
end
My logger.log file is in the same directory. Now, when I run: $ ruby logger.rb I see the last 10 lines from my log, but when I open logger.log file append some log data to it, the console doesn't show any progress. I mean it doesn't output the new log I appended.
I thought there may be in an issue with this. So, I tried inheriting and including the File::Tail in the inherited class, like this:
$:.unshift File.dirname(__FILE__)
filename = 'logger.log'
require "file-tail"
class FileTail < File
require "file-tail"
include File::Tail
end
log = FileTail.new(filename)
log.interval = 10
log.backward(10)
log.tail { |line| print line }
However this behaves the same way!!
Any pointers?
I am running on MAC OC X 10.8.5 with ruby-2.0.0-p353 installed.
Also, please let me know if anybody has implemented web version of tail in Ruby?
My Bad. This works when I closed all streams of my logger file. I'd opened the file in my ruby code but, didn't close the file stream. Maybe that's why I didn't see any log output on my console using the file-tail.
So, make sure you close all streams of the log/text file you're running with file-tail program.
I'm writing some code which takes a file, passes that file to one of several binaries for processing, and monitors the conversion process for errors. I've written and tested the following routine on OSX but linux fails for reasons about which I'm not clear.
#run the command, capture the output so it doesn't display
PTY.spawn(command) {|r,w,pid|
until r.eof? do
##mark
puts r.readline
end
}
The command that runs varies quite a lot and the code at the ##mark has been simplified into a local echo in an attempt to debug the problem. The command executes and the script prints the expected output in the terminal and then throws an exception.
The error it produces on Debian systems is: Errno::EIO (Input/output error - /dev/pts/0):
All of the command strings I can come up with produce that error, and when I run the code without the local echo block it runs just fine:
PTY.spawn(command) {|r,w,pid|}
In either case the command itself executes fine, but it seems like debian linux isn't sending eof up the pty. The doc pages for PTY, and IO on ruby-doc don't seem to lend any aid here.
Any suggestions? Thanks.
-vox-
So I had to go as far as reading the C source for the PTY library to get really satisfied with what is going on here.
The Ruby PTY doc doesn't really say what the comments in the source code say.
My solution was to put together a wrapper method and to call that from my script where needed. I've also boxed into the method waiting on the process to for sure exit and the accessing of the exit status from $?:
# file: lib/safe_pty.rb
require 'pty'
module SafePty
def self.spawn command, &block
PTY.spawn(command) do |r,w,p|
begin
yield r,w,p
rescue Errno::EIO
ensure
Process.wait p
end
end
$?.exitstatus
end
end
This is used basically the same as PTY.spawn:
require 'safe_pty'
exit_status = SafePty.spawn(command) do |r,w,pid|
until r.eof? do
logger.debug r.readline
end
end
#test exit_status for zeroness
I was more than a little frustrated to find out that this is a valid response, as it was completely undocumented on ruby-doc.
It seems valid for Errno::EIO to be raised here (it simply means the child process has finished and closed the stream), so you should expect that and catch it.
For example, see the selected answer in Continuously read from STDOUT of external process in Ruby and http://www.shanison.com/2010/09/11/ptychildexited-exception-and-ptys-exit-status/
BTW, I did some testing. On Ruby 1.8.7 on Ubuntu 10.04, I don't get a error. With Ruby 1.9.3, I do. With JRuby 1.6.4 on Ubuntu in both 1.8 and 1.9 modes, I don't get an error. On OS X, with 1.8.7, 1.9.2 and 1.9.3, I don't get an error. The behavior is obviously dependent on your Ruby version and platform.
As answered here and here, EIO can be avoided by keeping a file descriptor to the pty slave device open in the parent process.
Since PTY.spawn closes the slave file descriptor passed to the child process, a simple workaround is to open a new one. For example:
PTY.spawn("ls") do |r, w, pid|
r2 = File.open(r.path)
while IO.select([r], [], [], 1)
puts r.gets
end
r2.close
end
ruby-doc.org says this since ruby 1.9:
# The result of read operation when pty slave is closed is platform
# dependent.
ret = begin
m.gets # FreeBSD returns nil.
rescue Errno::EIO # GNU/Linux raises EIO.
nil
end
Ok, so now I get this behavior is "normal" on Linux, but that means it's a little tricky to get the output of a PTY. If you do m.read it reads everything and then throws it away and raises Errno::EIO. You really need to read the content chunk by chunk with m.readline. And even then you risk losing the last line if it doesn't end with "\n" for whatever reason. To be extra safe you need to read the content byte by byte with m.read(1)
Additional note about the effect of tty and pty on buffering: it's not the same as STDOUT.sync = true (unbuffered output) in the child process, but rather it triggers line buffering, where output is flushed on "\n"
I want to write a Ruby program that will always be running in the background (a daemon) on my Mac.
Can someone point me in the right direction on how this would be done?
Ruby 1.9.x has now the following:
Process.daemon
Put it in your code and that's it.
Taken from "Daemon Processes in Ruby."
Use Daemonize.rb
require 'daemons'
Daemons.daemonize
Very simple sample: http://github.com/utkarsh2012/backitup/blob/master/backitup.rb
How to install daemons gem:
gem install daemons
Ah, Google to the rescue! Check out
http://fitzgeraldsteele.wordpress.com/2009/05/04/launchd-example-start-web-server-at-boot-time/
wherein a helpful blogger provides an example of writing a launchd plist to launch a ruby Web application server.
This is a module to daemonize your code. Here's an offshoot that wraps an existing script.
Essentially it boils down to this (from Travis Whitton's Daemonize.rb, the first link above, modified for some program I wrote ages ago):
private
# This method causes the current running process to become a daemon
# If closefd is true, all existing file descriptors are closed
def daemonize(pathStdErr, oldmode=0, closefd=false)
srand # Split rand streams between spawning and daemonized process
safefork and exit# Fork and exit from the parent
# Detach from the controlling terminal
unless sess_id = Process.setsid
raise 'Cannot detach from controlled terminal'
end
# Prevent the possibility of acquiring a controlling terminal
if oldmode.zero?
trap 'SIGHUP', 'IGNORE'
exit if pid = safefork
end
Dir.chdir "/" # Release old working directory
File.umask 0000 # Insure sensible umask
if closefd
# Make sure all file descriptors are closed
ObjectSpace.each_object(IO) do |io|
unless [STDIN, STDOUT, STDERR].include?(io)
io.close rescue nil
end
end
end
STDIN.reopen "/dev/null" # Free file descriptors and
STDOUT.reopen "/dev/null" # point them somewhere sensible
STDERR.reopen pathStdErr, "w" # STDOUT/STDERR should go to a logfile
return oldmode ? sess_id : 0 # Return value is mostly irrelevant
end
# Try to fork if at all possible retrying every 5 sec if the
# maximum process limit for the system has been reached
def safefork
tryagain = true
while tryagain
tryagain = false
begin
if pid = fork
return pid
end
rescue Errno::EWOULDBLOCK
sleep 5
tryagain = true
end
end
end
Need to see the daemons-rails gem for Rails 3 (based on rails_generator):
https://github.com/mirasrael/daemons-rails
Possible to generate daemon stub like this:
rails generate daemon <name>
Features:
individual control script per daemon
rake:daemon command per daemon
capistrano friendly
app-wide control script
monitoring API
possible multiple daemon sets