I've been using resque on Heroku, which will from time to time interrupt your jobs with a SIGTERM.
Thus far I've handled this with a simple:
def process(options)
do_the_job
rescue Resque::TermException
self.defer options
end
We've started using resque-status so that we can keep track of jobs, but the method above obviously breaks that as the job will show completed when actually it's been deferred to another job.
My current thinking is that instead of deferring the current job in resque, there needs to be another job that re-queues jobs that have failed due to SIGTERM.
The trick comes in that some jobs are more complicated:
def process(options)
do_part1 unless options['part1_finished']
options['part1_finished']
do_part2
rescue Resque::TermException
self.defer options
end
Simply removing the rescue and simply retrying those jobs would cause an exception when do_part1 gets repeated.
Looking more deeply into how resque-status works, a possible work around is to go straight to resque for the re-queue using the same parameters that resque-status would use.
def process
do_part1 unless options['part1_finished']
options['part1_finished']
do_part2
rescue Resque::TermException
Resque.enqueue self.class, uuid, options
raise DeferredToNewJob
end
Of course, this is undocumented so may be incompatible with future releases of resque-status.
There is a draw back: between that job failing and the new job picking it up, the status of the first job will be reported by resque-status.
This is why I re-raise a new exception - otherwise the job status will show completed until the new worker picks up the old job, which may confuse processes that are watching and waiting for the job to finish.
By raising a new exception DeferredToNewJob, the job status will temporarily show failure, which is easier to work around at the front end, and the specific exception can be automatically cleared from the resque failure queue.
UPDATE
resque-status provides support for on_failure handler. If a method with this name is defined as an instance method on the class, we can make this even simpler
Here's my on_failure
def on_failure(e)
if e.is_a? DeferredToNewJob
tick('Waiting for new job')
else
raise e
end
end
With this in place the job spends basically no time in the failed state for processes watching it's status.
In addition, if resque-status finds this handler, then it won't raise the exception up to resque, so it won't get added to the failed queue.
I have an ActiveJob that's supposed to load a piece of data from an external system over HTTP. When that job completes, I want to queue a second job that does some postprocessing and then submits the data to a different external system.
I don't want the first job to know about the second job, because
encapsulation
reusability
it's none of the first job's business, basically
Likewise, I don't want the first job to care what happens next if the data-loading fails -- maybe the user gets notified, maybe we retry after a timeout, maybe we just log it and throw up our hands -- again it could vary based on the details of the exception, and there's no need for the job to include the logic for that or the connections to other systems to handle it.
In Java (which is where I have the most experience), I could use something like Guava's ListenableFuture to add success and failure callbacks after the fact:
MyDataLoader loader = new MyDataLoader(someDataSource)
ListenableFuture<Data> future = executor.submit(loader);
Futures.addCallback(future, new FutureCallback<Data>() {
public void onSuccess(Data result) {
processData(result);
}
public void onFailure(Throwable t) {
handleFailure(t);
}
});
ActiveJob, though, doesn't seem to provide this sort of external callback mechanism -- as best I can make out from relevant sections in "Active Job Basics", after_perform and rescue_from are only meant to be called from within the job class. And after_peform isn't meant to distinguish between success and failure.
So the best I've been able to come up with (and I'm not claiming it's very good) is to pass a couple of lambdas into the job's perform method, thus:
class MyRecordLoader < ActiveJob::Base
# Loads data expensively (hopefully on a background queue) and passes
# the result, or any exception, to the appropriate specified lambda.
#
# #param data_source [String] the URL to load data from
# #param on_success [-> (String)] A lambda that will be passed the record
# data, if it's loaded successfully
# #param on_failure [-> (Exception)] A lambda that will be passed any
# exception, if there is one
def perform(data_source, on_success, on_failure)
begin
result = load_data_expensively_from data_source
on_success.call(result)
rescue => exception
on_failure.call(exception)
end
end
end
(Side note: I have no idea what the yardoc syntax is for declaring lambdas as parameters. Does this look correct, or, failing that, plausible?)
The caller would then have to pass these in:
MyRecordLoader.perform_later(
some_data_source,
method(:process_data),
method(:handle_failure)
)
That's not terrible, at least on the calling side, but it seems clunky, and I can't help but suspect there's a common pattern for this that I'm just not finding. And I'm somewhat concerned that, as a Ruby/Rails novice, I'm just bending ActiveJob to do something it was never meant to do in the first place. All the ActiveJob examples I'm finding are 'fire and forget' -- asynchronously "returning" a result doesn't seem to be an ActiveJob use case.
Also, it's not clear to me that this will work at all in the case of a back-end like Resque that runs the jobs in a separate process.
What's "the Ruby way" to do this?
Update: As hinted at by dre-hh, ActiveJob turned out not to be the right tool here. It was also unreliable, and overcomplicated for the situation. I switched to Concurrent Ruby instead, which fits the use case better, and which, since the tasks are mostly IO-bound, is fast enough even on MRI, despite the GIL.
ActiveJob is not an async Library like a future or promise.
It is just an interface for performing tasks in a background. The current thread/process receives no result of this operation.
For example when using Sidekiq as ActiveJob queue, it will serialize the parameters of the perform method into the redis store. Another daemon process running within the context of your rails app will be watching the redis queue and instantiate your worker with the serialized data.
So passing callbacks might be alright, however why having them as methods on another class. Passing callbacks would make sense if those are dynamic (changing on different invocation). However as you have them implemented on the calling class, consider just moving those methods into your job worker class.
I am trying to make calls to an external API. I handle four or more exceptions for the call.
If I make multiple calls, the code increases very quick. Should I be writing a wrapper for each such call which handles the exceptions and returns data?
Here is an example of such code (this is not mine). The call to user_search is followed by the exception handling.
Note: I am not using Rails
begin
#twitter = Twitter.user_search(name)
rescue Twitter::Unauthorized
puts "Not authorized. Please check the Twitter credentials at the top of the script."
break
rescue Twitter::BadRequest => e
puts "Hit rate limit. Continuing scraping at #{e.ratelimit_reset}"
sleep e.retry_after
retry
rescue Exception => e
puts "Something else went wrong:"
puts e.message
end
I've changed the title of the question. I think the issue is more how to handle long exception handling code. In the example code suppose I have multiple calls to the twitter API followed by exception handling, it seems like the exception handling code disrupts reading the code which does the actual work.
Write your exception handler around a chunk of code so that, if the first line in the block fails, you're happy skipping all the code up until the last line in the block.
If an exception invalidates the whole rest of the method that the handler appears in, consider letting the exception bubble up to the next layer. Not everything necessarily has to be caught by your method.
Often I find myself writing exception handlers around single lines of code (with suitable recovery code) but it's not a rule.
def perform
refund_log = {
success: refund_retry.success?,
amount: refund_amount,
action: "refund"
}
if refund_retry.success?
refund_log[:reference] = refund_retry.transaction.id
refund_log[:message] = refund_retry.transaction.status
else
refund_log[:message] = refund_retry.message
refund_log[:params] = {}
refund_retry.errors.each do |error|
refund_log[:params][error.code] = error.message
end
order_transaction.message = refund_log[:params].values.join('|')
raise "delayed RefundJob has failed"
end
end
When I raise "delayed RefundJob has failed" in the else statement, it creates an Airbrake. I want to run the job again if it ends up in the else section.
Is there any way to re-queue the job without raising an exception? And prevent creating an airbrake?
I am using delayed_job version 1.
The cleanest way would be to re-queue, i.e. create a new job and enqueue it, and then exit the method normally.
To elaborate on #Roman's response, you can create a new job, with a retry parameter in it, and enqueue it.
If you maintain the retry parameter (increment it each time you re-enqueue a job), you can track how many retries you made, and thus avoid an endless retry loop.
DelayedJob expects a job to raise an error to requeued, by definition.
From there you can either :
Ignore your execpetion on airbrake side, see https://github.com/airbrake/airbrake#filtering so it still gets queued again without filling your logs
Dive into DelayedJob code where you can see on https://github.com/tobi/delayed_job/blob/master/lib/delayed/job.rb#L65 that a method named reschedule is available and used by run_with_lock ( https://github.com/tobi/delayed_job/blob/master/lib/delayed/job.rb#L99 ). From there you can call reschedule it manually, instead of raising your exception.
About the later solution, I advise adding some mechanism that still fill an airbrake report on the third or later try, you can still detect that something is wrong without the hassle of having your logs filled by the attempts.
Ryan Davis’s Ruby QuickRef says (without explanation):
Don’t rescue Exception. EVER. or I will stab you.
Why not? What’s the right thing to do?
TL;DR: Use StandardError instead for general exception catching. When the original exception is re-raised (e.g. when rescuing to log the exception only), rescuing Exception is probably okay.
Exception is the root of Ruby's exception hierarchy, so when you rescue Exception you rescue from everything, including subclasses such as SyntaxError, LoadError, and Interrupt.
Rescuing Interrupt prevents the user from using CTRLC to exit the program.
Rescuing SignalException prevents the program from responding correctly to signals. It will be unkillable except by kill -9.
Rescuing SyntaxError means that evals that fail will do so silently.
All of these can be shown by running this program, and trying to CTRLC or kill it:
loop do
begin
sleep 1
eval "djsakru3924r9eiuorwju3498 += 5u84fior8u8t4ruyf8ihiure"
rescue Exception
puts "I refuse to fail or be stopped!"
end
end
Rescuing from Exception isn't even the default. Doing
begin
# iceberg!
rescue
# lifeboats
end
does not rescue from Exception, it rescues from StandardError. You should generally specify something more specific than the default StandardError, but rescuing from Exception broadens the scope rather than narrowing it, and can have catastrophic results and make bug-hunting extremely difficult.
If you have a situation where you do want to rescue from StandardError and you need a variable with the exception, you can use this form:
begin
# iceberg!
rescue => e
# lifeboats
end
which is equivalent to:
begin
# iceberg!
rescue StandardError => e
# lifeboats
end
One of the few common cases where it’s sane to rescue from Exception is for logging/reporting purposes, in which case you should immediately re-raise the exception:
begin
# iceberg?
rescue Exception => e
# do some logging
raise # not enough lifeboats ;)
end
The real rule is: Don't throw away exceptions. The objectivity of the author of your quote is questionable, as evidenced by the fact that it ends with
or I will stab you
Of course, be aware that signals (by default) throw exceptions, and normally long-running processes are terminated through a signal, so catching Exception and not terminating on signal exceptions will make your program very hard to stop. So don't do this:
#! /usr/bin/ruby
while true do
begin
line = STDIN.gets
# heavy processing
rescue Exception => e
puts "caught exception #{e}! ohnoes!"
end
end
No, really, don't do it. Don't even run that to see if it works.
However, say you have a threaded server and you want all exceptions to not:
be ignored (the default)
stop the server (which happens if you say thread.abort_on_exception = true).
Then this is perfectly acceptable in your connection handling thread:
begin
# do stuff
rescue Exception => e
myLogger.error("uncaught #{e} exception while handling connection: #{e.message}")
myLogger.error("Stack trace: #{backtrace.map {|l| " #{l}\n"}.join}")
end
The above works out to a variation of Ruby's default exception handler, with the advantage that it doesn't also kill your program. Rails does this in its request handler.
Signal exceptions are raised in the main thread. Background threads won't get them, so there is no point in trying to catch them there.
This is particularly useful in a production environment, where you do not want your program to simply stop whenever something goes wrong. Then you can take the stack dumps in your logs and add to your code to deal with specific exception further down the call chain and in a more graceful manner.
Note also that there is another Ruby idiom which has much the same effect:
a = do_something rescue "something else"
In this line, if do_something raises an exception, it is caught by Ruby, thrown away, and a is assigned "something else".
Generally, don't do that, except in special cases where you know you don't need to worry. One example:
debugger rescue nil
The debugger function is a rather nice way to set a breakpoint in your code, but if running outside a debugger, and Rails, it raises an exception. Now theoretically you shouldn't be leaving debug code lying around in your program (pff! nobody does that!) but you might want to keep it there for a while for some reason, but not continually run your debugger.
Note:
If you've run someone else's program that catches signal exceptions and ignores them, (say the code above) then:
in Linux, in a shell, type pgrep ruby, or ps | grep ruby, look for your offending program's PID, and then run kill -9 <PID>.
in Windows, use the Task Manager (CTRL-SHIFT-ESC), go to the "processes" tab, find your process, right click it and select "End process".
If you are working with someone else's program which is, for whatever reason, peppered with these ignore-exception blocks, then putting this at the top of the mainline is one possible cop-out:
%W/INT QUIT TERM/.each { |sig| trap sig,"SYSTEM_DEFAULT" }
This causes the program to respond to the normal termination signals by immediately terminating, bypassing exception handlers, with no cleanup. So it could cause data loss or similar. Be careful!
If you need to do this:
begin
do_something
rescue Exception => e
critical_cleanup
raise
end
you can actually do this:
begin
do_something
ensure
critical_cleanup
end
In the second case, critical cleanup will be called every time, whether or not an exception is thrown.
TL;DR
Don't rescue Exception => e (and not re-raise the exception) - or you might drive off a bridge.
Let's say you are in a car (running Ruby). You recently installed a new steering wheel with the over-the-air upgrade system (which uses eval), but you didn't know one of the programmers messed up on syntax.
You are on a bridge, and realize you are going a bit towards the railing, so you turn left.
def turn_left
self.turn left:
end
oops! That's probably Not Good™, luckily, Ruby raises a SyntaxError.
The car should stop immediately - right?
Nope.
begin
#...
eval self.steering_wheel
#...
rescue Exception => e
self.beep
self.log "Caught #{e}.", :warn
self.log "Logged Error - Continuing Process.", :info
end
beep beep
Warning: Caught SyntaxError Exception.
Info: Logged Error - Continuing Process.
You notice something is wrong, and you slam on the emergency breaks (^C: Interrupt)
beep beep
Warning: Caught Interrupt Exception.
Info: Logged Error - Continuing Process.
Yeah - that didn't help much. You're pretty close to the rail, so you put the car in park (killing: SignalException).
beep beep
Warning: Caught SignalException Exception.
Info: Logged Error - Continuing Process.
At the last second, you pull out the keys (kill -9), and the car stops, you slam forward into the steering wheel (the airbag can't inflate because you didn't gracefully stop the program - you terminated it), and the computer in the back of your car slams into the seat in front of it. A half-full can of Coke spills over the papers. The groceries in the back are crushed, and most are covered in egg yolk and milk. The car needs serious repair and cleaning. (Data Loss)
Hopefully you have insurance (Backups). Oh yeah - because the airbag didn't inflate, you're probably hurt (getting fired, etc).
But wait! There's more reasons why you might want to use rescue Exception => e!
Let's say you're that car, and you want to make sure the airbag inflates if the car is exceeding its safe stopping momentum.
begin
# do driving stuff
rescue Exception => e
self.airbags.inflate if self.exceeding_safe_stopping_momentum?
raise
end
Here's the exception to the rule: You can catch Exception only if you re-raise the exception. So, a better rule is to never swallow Exception, and always re-raise the error.
But adding rescue is both easy to forget in a language like Ruby, and putting a rescue statement right before re-raising an issue feels a little non-DRY. And you do not want to forget the raise statement. And if you do, good luck trying to find that error.
Thankfully, Ruby is awesome, you can just use the ensure keyword, which makes sure the code runs. The ensure keyword will run the code no matter what - if an exception is thrown, if one isn't, the only exception being if the world ends (or other unlikely events).
begin
# do driving stuff
ensure
self.airbags.inflate if self.exceeding_safe_stopping_momentum?
end
Boom! And that code should run anyways. The only reason you should use rescue Exception => e is if you need access to the exception, or if you only want code to run on an exception. And remember to re-raise the error. Every time.
Note: As #Niall pointed out, ensure always runs. This is good because sometimes your program can lie to you and not throw exceptions, even when issues occur. With critical tasks, like inflating airbags, you need to make sure it happens no matter what. Because of this, checking every time the car stops, whether an exception is thrown or not, is a good idea. Even though inflating airbags is a bit of an uncommon task in most programming contexts, this is actually pretty common with most cleanup tasks.
Because this captures all exceptions. It's unlikely that your program can recover from any of them.
You should handle only exceptions that you know how to recover from. If you don't anticipate a certain kind of exception, don't handle it, crash loudly (write details to the log), then diagnose logs and fix code.
Swallowing exceptions is bad, don't do this.
That's a specific case of the rule that you shouldn't catch any exception you don't know how to handle. If you don't know how to handle it, it's always better to let some other part of the system catch and handle it.
This blog post explains it perfectly:
Ruby's Exception vs StandardError: What's the difference?
Why you shouldn't rescue Exception
The problem with rescuing Exception
is that it actually rescues every exception that inherits from
Exception. Which is....all of them!
That's a problem because there are some exceptions that are used
internally by Ruby. They don't have anything to do with your app, and
swallowing them will cause bad things to happen.
Here are a few of the big ones:
SignalException::Interrupt - If you rescue this, you can't exit your
app by hitting control-c.
ScriptError::SyntaxError - Swallowing syntax errors means that things
like puts("Forgot something) will fail silently.
NoMemoryError - Wanna know what happens when your program keeps
running after it uses up all the RAM? Me neither.
begin
do_something()
rescue Exception => e
# Don't do this. This will swallow every single exception. Nothing gets past it.
end
I'm guessing that you don't really want to swallow any of these
system-level exceptions. You only want to catch all of your
application level errors. The exceptions caused YOUR code.
Luckily, there's an easy way to to this.
Rescue StandardError Instead
All of the exceptions that you should care about inherit from StandardError. These are our old friends:
NoMethodError - raised when you try to invoke a method that doesn't exist
TypeError - caused by things like 1 + ""
RuntimeError - who could forget good old RuntimeError?
To rescue errors like these, you'll want to rescue StandardError. You COULD do it by writing something like this:
begin
do_something()
rescue StandardError => e
# Only your app's exceptions are swallowed. Things like SyntaxErrror are left alone.
end
But Ruby has made it much easier for use.
When you don't specify an exception class at all, ruby assumes you mean StandardError. So the code below is identical to the above code:
begin
do_something()
rescue => e
# This is the same as rescuing StandardError
end