How to catch top level failures on an EventMachine server? - ruby

I have an EventMachine server that I am monitoring with monit. Sometimes it crashes, and I am trying to figure out why, but it is unclear to me how I can just log all top level failures. I tried code like this:
begin
EventMachine::run do
EventMachine::start_server('0.0.0.0', PORT, MyServer)
end
rescue Exception => e
puts "FAILURE: #{e.class}: #{e}"
end
but that does not seem to ever catch errors. I suspect it might be something like running out of memory, which I am tracking separately, but still I would like this server to log its proximate cause of failure if possible.

If you want a catch-all error handler, try EM.error_handler. Example from the docs:
EM.error_handler{ |e|
puts "Error raised during event loop: #{e.message}"
}
You may also want more fine-grained error handling, in which case you can use the errback mechanism (see Deferrable). So for example you could have in your reactor loop:
EventMachine::run do
server = EventMachine::start_server('0.0.0.0', PORT, MyServer)
server.errback { # handle error thrown by server here }
end
For this to work, include Deferrable in your MyServer, then whenever you want to raise an error, call fail.

Related

Handling Errno::ETIMEDOUT exception in capistrano

I am using a block to dynamically generate a list of servers I want to push code to. Occasionally a few of the machines are unreachable, resulting in a "Errno::ETIMEDOUT". This stop the entire deployment process. How do I rescue this and run some custom code using capistrano?
You'd have to wrap your server list generator with a try catch block.
begin
# ... Get server list
rescue Errno::ETIMEDOUT
# ... Handle cases when connection times out
end

rake still aborts on RuntimeError even though exception is rescued

I am working on a rake system, and I have some new code which needs to do some consistency checking. I would like the code to not exit on the first error; I would like it to continue to finish checks on the rest of the data. If it does hit a consistency error, it can throw an exception. I am trying to use a simple construct just to catch RuntimeError:
begin
<code which may throw RuntimeError...>
rescue RuntimeError => e
<print info about exception ...>
end
This seems to execute the rescue block; the error message from the rescue is printed. The problem is that rake still aborts, even though the exception has been caught and handled. Is there some setting in rake which sets abort_on_exception to true? Is there any way yo change this? I do not want it to abort on this exception, I just want the error printed, and to continue with the rest of the checks. It will exit on a higher level if the complete consistency check reported an error. Perhaps I need to throw something other than a RuntimeError? I am just using raise with a string argument.
Can you try Exception insted of RuntimeError exception
begin
<code which may throw RuntimeError...>
rescue Exception => e
p e.message
p e.backtrace
end

Why is it bad style to `rescue Exception => e` in Ruby?

Ryan Davis’s Ruby QuickRef says (without explanation):
Don’t rescue Exception. EVER. or I will stab you.
Why not? What’s the right thing to do?
TL;DR: Use StandardError instead for general exception catching. When the original exception is re-raised (e.g. when rescuing to log the exception only), rescuing Exception is probably okay.
Exception is the root of Ruby's exception hierarchy, so when you rescue Exception you rescue from everything, including subclasses such as SyntaxError, LoadError, and Interrupt.
Rescuing Interrupt prevents the user from using CTRLC to exit the program.
Rescuing SignalException prevents the program from responding correctly to signals. It will be unkillable except by kill -9.
Rescuing SyntaxError means that evals that fail will do so silently.
All of these can be shown by running this program, and trying to CTRLC or kill it:
loop do
begin
sleep 1
eval "djsakru3924r9eiuorwju3498 += 5u84fior8u8t4ruyf8ihiure"
rescue Exception
puts "I refuse to fail or be stopped!"
end
end
Rescuing from Exception isn't even the default. Doing
begin
# iceberg!
rescue
# lifeboats
end
does not rescue from Exception, it rescues from StandardError. You should generally specify something more specific than the default StandardError, but rescuing from Exception broadens the scope rather than narrowing it, and can have catastrophic results and make bug-hunting extremely difficult.
If you have a situation where you do want to rescue from StandardError and you need a variable with the exception, you can use this form:
begin
# iceberg!
rescue => e
# lifeboats
end
which is equivalent to:
begin
# iceberg!
rescue StandardError => e
# lifeboats
end
One of the few common cases where it’s sane to rescue from Exception is for logging/reporting purposes, in which case you should immediately re-raise the exception:
begin
# iceberg?
rescue Exception => e
# do some logging
raise # not enough lifeboats ;)
end
The real rule is: Don't throw away exceptions. The objectivity of the author of your quote is questionable, as evidenced by the fact that it ends with
or I will stab you
Of course, be aware that signals (by default) throw exceptions, and normally long-running processes are terminated through a signal, so catching Exception and not terminating on signal exceptions will make your program very hard to stop. So don't do this:
#! /usr/bin/ruby
while true do
begin
line = STDIN.gets
# heavy processing
rescue Exception => e
puts "caught exception #{e}! ohnoes!"
end
end
No, really, don't do it. Don't even run that to see if it works.
However, say you have a threaded server and you want all exceptions to not:
be ignored (the default)
stop the server (which happens if you say thread.abort_on_exception = true).
Then this is perfectly acceptable in your connection handling thread:
begin
# do stuff
rescue Exception => e
myLogger.error("uncaught #{e} exception while handling connection: #{e.message}")
myLogger.error("Stack trace: #{backtrace.map {|l| " #{l}\n"}.join}")
end
The above works out to a variation of Ruby's default exception handler, with the advantage that it doesn't also kill your program. Rails does this in its request handler.
Signal exceptions are raised in the main thread. Background threads won't get them, so there is no point in trying to catch them there.
This is particularly useful in a production environment, where you do not want your program to simply stop whenever something goes wrong. Then you can take the stack dumps in your logs and add to your code to deal with specific exception further down the call chain and in a more graceful manner.
Note also that there is another Ruby idiom which has much the same effect:
a = do_something rescue "something else"
In this line, if do_something raises an exception, it is caught by Ruby, thrown away, and a is assigned "something else".
Generally, don't do that, except in special cases where you know you don't need to worry. One example:
debugger rescue nil
The debugger function is a rather nice way to set a breakpoint in your code, but if running outside a debugger, and Rails, it raises an exception. Now theoretically you shouldn't be leaving debug code lying around in your program (pff! nobody does that!) but you might want to keep it there for a while for some reason, but not continually run your debugger.
Note:
If you've run someone else's program that catches signal exceptions and ignores them, (say the code above) then:
in Linux, in a shell, type pgrep ruby, or ps | grep ruby, look for your offending program's PID, and then run kill -9 <PID>.
in Windows, use the Task Manager (CTRL-SHIFT-ESC), go to the "processes" tab, find your process, right click it and select "End process".
If you are working with someone else's program which is, for whatever reason, peppered with these ignore-exception blocks, then putting this at the top of the mainline is one possible cop-out:
%W/INT QUIT TERM/.each { |sig| trap sig,"SYSTEM_DEFAULT" }
This causes the program to respond to the normal termination signals by immediately terminating, bypassing exception handlers, with no cleanup. So it could cause data loss or similar. Be careful!
If you need to do this:
begin
do_something
rescue Exception => e
critical_cleanup
raise
end
you can actually do this:
begin
do_something
ensure
critical_cleanup
end
In the second case, critical cleanup will be called every time, whether or not an exception is thrown.
TL;DR
Don't rescue Exception => e (and not re-raise the exception) - or you might drive off a bridge.
Let's say you are in a car (running Ruby). You recently installed a new steering wheel with the over-the-air upgrade system (which uses eval), but you didn't know one of the programmers messed up on syntax.
You are on a bridge, and realize you are going a bit towards the railing, so you turn left.
def turn_left
self.turn left:
end
oops! That's probably Not Good™, luckily, Ruby raises a SyntaxError.
The car should stop immediately - right?
Nope.
begin
#...
eval self.steering_wheel
#...
rescue Exception => e
self.beep
self.log "Caught #{e}.", :warn
self.log "Logged Error - Continuing Process.", :info
end
beep beep
Warning: Caught SyntaxError Exception.
Info: Logged Error - Continuing Process.
You notice something is wrong, and you slam on the emergency breaks (^C: Interrupt)
beep beep
Warning: Caught Interrupt Exception.
Info: Logged Error - Continuing Process.
Yeah - that didn't help much. You're pretty close to the rail, so you put the car in park (killing: SignalException).
beep beep
Warning: Caught SignalException Exception.
Info: Logged Error - Continuing Process.
At the last second, you pull out the keys (kill -9), and the car stops, you slam forward into the steering wheel (the airbag can't inflate because you didn't gracefully stop the program - you terminated it), and the computer in the back of your car slams into the seat in front of it. A half-full can of Coke spills over the papers. The groceries in the back are crushed, and most are covered in egg yolk and milk. The car needs serious repair and cleaning. (Data Loss)
Hopefully you have insurance (Backups). Oh yeah - because the airbag didn't inflate, you're probably hurt (getting fired, etc).
But wait! There's more reasons why you might want to use rescue Exception => e!
Let's say you're that car, and you want to make sure the airbag inflates if the car is exceeding its safe stopping momentum.
begin
# do driving stuff
rescue Exception => e
self.airbags.inflate if self.exceeding_safe_stopping_momentum?
raise
end
Here's the exception to the rule: You can catch Exception only if you re-raise the exception. So, a better rule is to never swallow Exception, and always re-raise the error.
But adding rescue is both easy to forget in a language like Ruby, and putting a rescue statement right before re-raising an issue feels a little non-DRY. And you do not want to forget the raise statement. And if you do, good luck trying to find that error.
Thankfully, Ruby is awesome, you can just use the ensure keyword, which makes sure the code runs. The ensure keyword will run the code no matter what - if an exception is thrown, if one isn't, the only exception being if the world ends (or other unlikely events).
begin
# do driving stuff
ensure
self.airbags.inflate if self.exceeding_safe_stopping_momentum?
end
Boom! And that code should run anyways. The only reason you should use rescue Exception => e is if you need access to the exception, or if you only want code to run on an exception. And remember to re-raise the error. Every time.
Note: As #Niall pointed out, ensure always runs. This is good because sometimes your program can lie to you and not throw exceptions, even when issues occur. With critical tasks, like inflating airbags, you need to make sure it happens no matter what. Because of this, checking every time the car stops, whether an exception is thrown or not, is a good idea. Even though inflating airbags is a bit of an uncommon task in most programming contexts, this is actually pretty common with most cleanup tasks.
Because this captures all exceptions. It's unlikely that your program can recover from any of them.
You should handle only exceptions that you know how to recover from. If you don't anticipate a certain kind of exception, don't handle it, crash loudly (write details to the log), then diagnose logs and fix code.
Swallowing exceptions is bad, don't do this.
That's a specific case of the rule that you shouldn't catch any exception you don't know how to handle. If you don't know how to handle it, it's always better to let some other part of the system catch and handle it.
This blog post explains it perfectly:
Ruby's Exception vs StandardError: What's the difference?
Why you shouldn't rescue Exception
The problem with rescuing Exception
is that it actually rescues every exception that inherits from
Exception. Which is....all of them!
That's a problem because there are some exceptions that are used
internally by Ruby. They don't have anything to do with your app, and
swallowing them will cause bad things to happen.
Here are a few of the big ones:
SignalException::Interrupt - If you rescue this, you can't exit your
app by hitting control-c.
ScriptError::SyntaxError - Swallowing syntax errors means that things
like puts("Forgot something) will fail silently.
NoMemoryError - Wanna know what happens when your program keeps
running after it uses up all the RAM? Me neither.
begin
do_something()
rescue Exception => e
# Don't do this. This will swallow every single exception. Nothing gets past it.
end
I'm guessing that you don't really want to swallow any of these
system-level exceptions. You only want to catch all of your
application level errors. The exceptions caused YOUR code.
Luckily, there's an easy way to to this.
Rescue StandardError Instead
All of the exceptions that you should care about inherit from StandardError. These are our old friends:
NoMethodError - raised when you try to invoke a method that doesn't exist
TypeError - caused by things like 1 + ""
RuntimeError - who could forget good old RuntimeError?
To rescue errors like these, you'll want to rescue StandardError. You COULD do it by writing something like this:
begin
do_something()
rescue StandardError => e
# Only your app's exceptions are swallowed. Things like SyntaxErrror are left alone.
end
But Ruby has made it much easier for use.
When you don't specify an exception class at all, ruby assumes you mean StandardError. So the code below is identical to the above code:
begin
do_something()
rescue => e
# This is the same as rescuing StandardError
end

Ruby Thread with "watchdog"

I'm implementing a ruby server for handling sockets being created from GPRS modules. The thing is that when the module powers down, there's no indication that the socket closed.
I'm doing threads to handle multiple sockets with the same server. What I'm asking is this: Is there a way to use a timer inside a thread, reset it after every socket input, and that if it hits the timeout, closes the thread? Where can I find more information about this?
EDIT: Code example that doesn't detect the socket closing
require 'socket'
server = TCPServer.open(41000)
loop do
Thread.start(server.accept) do |client|
puts "Client connected"
begin
loop do
line = client.readline
open('log.txt', 'a') { |f|
f.puts line.strip
}
end
rescue
puts "Client disconnected"
end
end
end
I think you need a heartbeat mechanism.
At a guess, your sockets are inexplably closing because you're not catching exceptions that are raised when they are closed by the remote end.
you need to wrap the connection handler in an exception catching block.
Without knowing what module/model you're using I will just fudge it and say you have a process_connection routine. So you need to do something like this:
def process_connection(conn)
begin
# do stuff
rescue Exception => e
STDERR.print "Caught exception #{e}: #{e.message}\n#{e.backtrace}\n"
ensure
conn.close
end
end
This will catch all exceptions and dump them to stderr with a stack trace. From there you can see what is causing them, and possibly handle them more gracefully elsewhere.
Just check the standar API Timeout:
require 'timeout'
status = Timeout::timeout(3){sleep(1)}
puts status.inspect
status = Timeout::timeout(1){sleep(2)}

'Who's online?' Ruby Network Program

I have several embedded linux systems that I want to write a 'Who's Online?' network service in Ruby. Below is related part of my code:
mySocket = UDPSocket.new
mySocket.bind("<broadcast>", 50050)
loop do
begin
text, sender = mySocket.recvfrom(1024)
puts text
if text =~ /KNOCK KNOCK/ then
begin
sock = UDPSocket.open
sock.send(r.ipaddress, 0, sender[3], 50051)
sock.close
rescue
retry
end
end
rescue Exception => inLoopEx
puts inLoopEx.message
puts inLoopEx.backtrace.inspect
retry
end
end
I send the 'KNOCK KNOCK' command from a PC. Now, the problem is since they all receive the message at the same time, they try to respond at the same time too, which causes a Broken Pipe exception (which is the reason of my 'rescue retry' code). This code works OK sometimes but; other times the rescue retry part of the code (which is waked by Broken Pipe exception from sock.send) causes one or more systems to respond after 5 seconds or so.
Is there a better way of doing this since I assume I cant escape the Broken Pipe exception?
I have found that exception was caused by the 'r.ipaddress' part in the send command, which is related to my embedded system's internals...

Resources