I'm using rspotify to gather a list of data from albums names. All the while I've reached Spotify's api rate limit and would now like to create a few fallbacks to wait until the I can search and retry the search in order to not lose the (x) amount of data I've already retrieved.
The gem uses RestClient but the unfortunately when I reach the rate limit I don't get the amount of time needed to wait until I can make another call:
.rvm/gems/ruby-2.5.1/gems/rest-client 2.0.2/lib/restclient/abstract_response.rb:223:in 'exception_with_response': 429 Too Many Requests (RestClient::TooManyRequests)
The above is all I'm given. The begin/rescue statement below doesn't work as when the code fails, it fails entirely without retrying. What am I doing wrong here?
begin
search = RSpotify::Album.search(album[:title])
rescue RestClient::ExceptionWithResponse, RestClient::TooManyRequests, Exception => e
puts e
retry
rescue e
puts e
retry
end
Here is how they suggest error handling:
https://github.com/rest-client/rest-client#response-callbacks-error-handling
I was thinking of maybe throttling, so inside the exception to use
RSpotify::authenticate("id", "token") with some of the multiple spotify accounts that I have
and then retry
So something like that all put together
begin
album = RSpotify::Album.find(track.first.album.id)
rescue RestClient::ExceptionWithResponse, RestClient::TooManyRequests, Exception => e
RSpotify::authenticate("id2", "token2")
retry
end
I am using a RubyGem (DeathByCaptcha) that makes HTTP calls to deathbycaptcha.com. Every so often the HTTP request times out or fails for some other unknown reason, and my Ruby scripts exits with an exception. I am trying to automate repeated instances of this method ("decode") and I am trying to determine if there is a way to prevent an error in this method from exiting the whole script.
EDIT: Since I'm bound to get flamed on here, I will mention upfront that the purpose of this is to determine the effectiveness of different captcha options on my website's registration page with common captcha-breakers, because I have had problems with spam signups.
Here is how to prevent the exception from exiting the script.
tries = 0
begin
# risky failing code
rescue
sleep(1) # sleep n seconds
tries += 1
retry if tries <= 3 # retry the risky code again
end
You would need to catch the exception that is raised and somehow handle it.
You are looking for something like
begin
# Send HTTP request
rescue WhateverExceptionClassYouGet > error
# Do something with the error
end
I am trying to make calls to an external API. I handle four or more exceptions for the call.
If I make multiple calls, the code increases very quick. Should I be writing a wrapper for each such call which handles the exceptions and returns data?
Here is an example of such code (this is not mine). The call to user_search is followed by the exception handling.
Note: I am not using Rails
begin
#twitter = Twitter.user_search(name)
rescue Twitter::Unauthorized
puts "Not authorized. Please check the Twitter credentials at the top of the script."
break
rescue Twitter::BadRequest => e
puts "Hit rate limit. Continuing scraping at #{e.ratelimit_reset}"
sleep e.retry_after
retry
rescue Exception => e
puts "Something else went wrong:"
puts e.message
end
I've changed the title of the question. I think the issue is more how to handle long exception handling code. In the example code suppose I have multiple calls to the twitter API followed by exception handling, it seems like the exception handling code disrupts reading the code which does the actual work.
Write your exception handler around a chunk of code so that, if the first line in the block fails, you're happy skipping all the code up until the last line in the block.
If an exception invalidates the whole rest of the method that the handler appears in, consider letting the exception bubble up to the next layer. Not everything necessarily has to be caught by your method.
Often I find myself writing exception handlers around single lines of code (with suitable recovery code) but it's not a rule.
Ryan Davis’s Ruby QuickRef says (without explanation):
Don’t rescue Exception. EVER. or I will stab you.
Why not? What’s the right thing to do?
TL;DR: Use StandardError instead for general exception catching. When the original exception is re-raised (e.g. when rescuing to log the exception only), rescuing Exception is probably okay.
Exception is the root of Ruby's exception hierarchy, so when you rescue Exception you rescue from everything, including subclasses such as SyntaxError, LoadError, and Interrupt.
Rescuing Interrupt prevents the user from using CTRLC to exit the program.
Rescuing SignalException prevents the program from responding correctly to signals. It will be unkillable except by kill -9.
Rescuing SyntaxError means that evals that fail will do so silently.
All of these can be shown by running this program, and trying to CTRLC or kill it:
loop do
begin
sleep 1
eval "djsakru3924r9eiuorwju3498 += 5u84fior8u8t4ruyf8ihiure"
rescue Exception
puts "I refuse to fail or be stopped!"
end
end
Rescuing from Exception isn't even the default. Doing
begin
# iceberg!
rescue
# lifeboats
end
does not rescue from Exception, it rescues from StandardError. You should generally specify something more specific than the default StandardError, but rescuing from Exception broadens the scope rather than narrowing it, and can have catastrophic results and make bug-hunting extremely difficult.
If you have a situation where you do want to rescue from StandardError and you need a variable with the exception, you can use this form:
begin
# iceberg!
rescue => e
# lifeboats
end
which is equivalent to:
begin
# iceberg!
rescue StandardError => e
# lifeboats
end
One of the few common cases where it’s sane to rescue from Exception is for logging/reporting purposes, in which case you should immediately re-raise the exception:
begin
# iceberg?
rescue Exception => e
# do some logging
raise # not enough lifeboats ;)
end
The real rule is: Don't throw away exceptions. The objectivity of the author of your quote is questionable, as evidenced by the fact that it ends with
or I will stab you
Of course, be aware that signals (by default) throw exceptions, and normally long-running processes are terminated through a signal, so catching Exception and not terminating on signal exceptions will make your program very hard to stop. So don't do this:
#! /usr/bin/ruby
while true do
begin
line = STDIN.gets
# heavy processing
rescue Exception => e
puts "caught exception #{e}! ohnoes!"
end
end
No, really, don't do it. Don't even run that to see if it works.
However, say you have a threaded server and you want all exceptions to not:
be ignored (the default)
stop the server (which happens if you say thread.abort_on_exception = true).
Then this is perfectly acceptable in your connection handling thread:
begin
# do stuff
rescue Exception => e
myLogger.error("uncaught #{e} exception while handling connection: #{e.message}")
myLogger.error("Stack trace: #{backtrace.map {|l| " #{l}\n"}.join}")
end
The above works out to a variation of Ruby's default exception handler, with the advantage that it doesn't also kill your program. Rails does this in its request handler.
Signal exceptions are raised in the main thread. Background threads won't get them, so there is no point in trying to catch them there.
This is particularly useful in a production environment, where you do not want your program to simply stop whenever something goes wrong. Then you can take the stack dumps in your logs and add to your code to deal with specific exception further down the call chain and in a more graceful manner.
Note also that there is another Ruby idiom which has much the same effect:
a = do_something rescue "something else"
In this line, if do_something raises an exception, it is caught by Ruby, thrown away, and a is assigned "something else".
Generally, don't do that, except in special cases where you know you don't need to worry. One example:
debugger rescue nil
The debugger function is a rather nice way to set a breakpoint in your code, but if running outside a debugger, and Rails, it raises an exception. Now theoretically you shouldn't be leaving debug code lying around in your program (pff! nobody does that!) but you might want to keep it there for a while for some reason, but not continually run your debugger.
Note:
If you've run someone else's program that catches signal exceptions and ignores them, (say the code above) then:
in Linux, in a shell, type pgrep ruby, or ps | grep ruby, look for your offending program's PID, and then run kill -9 <PID>.
in Windows, use the Task Manager (CTRL-SHIFT-ESC), go to the "processes" tab, find your process, right click it and select "End process".
If you are working with someone else's program which is, for whatever reason, peppered with these ignore-exception blocks, then putting this at the top of the mainline is one possible cop-out:
%W/INT QUIT TERM/.each { |sig| trap sig,"SYSTEM_DEFAULT" }
This causes the program to respond to the normal termination signals by immediately terminating, bypassing exception handlers, with no cleanup. So it could cause data loss or similar. Be careful!
If you need to do this:
begin
do_something
rescue Exception => e
critical_cleanup
raise
end
you can actually do this:
begin
do_something
ensure
critical_cleanup
end
In the second case, critical cleanup will be called every time, whether or not an exception is thrown.
TL;DR
Don't rescue Exception => e (and not re-raise the exception) - or you might drive off a bridge.
Let's say you are in a car (running Ruby). You recently installed a new steering wheel with the over-the-air upgrade system (which uses eval), but you didn't know one of the programmers messed up on syntax.
You are on a bridge, and realize you are going a bit towards the railing, so you turn left.
def turn_left
self.turn left:
end
oops! That's probably Not Good™, luckily, Ruby raises a SyntaxError.
The car should stop immediately - right?
Nope.
begin
#...
eval self.steering_wheel
#...
rescue Exception => e
self.beep
self.log "Caught #{e}.", :warn
self.log "Logged Error - Continuing Process.", :info
end
beep beep
Warning: Caught SyntaxError Exception.
Info: Logged Error - Continuing Process.
You notice something is wrong, and you slam on the emergency breaks (^C: Interrupt)
beep beep
Warning: Caught Interrupt Exception.
Info: Logged Error - Continuing Process.
Yeah - that didn't help much. You're pretty close to the rail, so you put the car in park (killing: SignalException).
beep beep
Warning: Caught SignalException Exception.
Info: Logged Error - Continuing Process.
At the last second, you pull out the keys (kill -9), and the car stops, you slam forward into the steering wheel (the airbag can't inflate because you didn't gracefully stop the program - you terminated it), and the computer in the back of your car slams into the seat in front of it. A half-full can of Coke spills over the papers. The groceries in the back are crushed, and most are covered in egg yolk and milk. The car needs serious repair and cleaning. (Data Loss)
Hopefully you have insurance (Backups). Oh yeah - because the airbag didn't inflate, you're probably hurt (getting fired, etc).
But wait! There's more reasons why you might want to use rescue Exception => e!
Let's say you're that car, and you want to make sure the airbag inflates if the car is exceeding its safe stopping momentum.
begin
# do driving stuff
rescue Exception => e
self.airbags.inflate if self.exceeding_safe_stopping_momentum?
raise
end
Here's the exception to the rule: You can catch Exception only if you re-raise the exception. So, a better rule is to never swallow Exception, and always re-raise the error.
But adding rescue is both easy to forget in a language like Ruby, and putting a rescue statement right before re-raising an issue feels a little non-DRY. And you do not want to forget the raise statement. And if you do, good luck trying to find that error.
Thankfully, Ruby is awesome, you can just use the ensure keyword, which makes sure the code runs. The ensure keyword will run the code no matter what - if an exception is thrown, if one isn't, the only exception being if the world ends (or other unlikely events).
begin
# do driving stuff
ensure
self.airbags.inflate if self.exceeding_safe_stopping_momentum?
end
Boom! And that code should run anyways. The only reason you should use rescue Exception => e is if you need access to the exception, or if you only want code to run on an exception. And remember to re-raise the error. Every time.
Note: As #Niall pointed out, ensure always runs. This is good because sometimes your program can lie to you and not throw exceptions, even when issues occur. With critical tasks, like inflating airbags, you need to make sure it happens no matter what. Because of this, checking every time the car stops, whether an exception is thrown or not, is a good idea. Even though inflating airbags is a bit of an uncommon task in most programming contexts, this is actually pretty common with most cleanup tasks.
Because this captures all exceptions. It's unlikely that your program can recover from any of them.
You should handle only exceptions that you know how to recover from. If you don't anticipate a certain kind of exception, don't handle it, crash loudly (write details to the log), then diagnose logs and fix code.
Swallowing exceptions is bad, don't do this.
That's a specific case of the rule that you shouldn't catch any exception you don't know how to handle. If you don't know how to handle it, it's always better to let some other part of the system catch and handle it.
This blog post explains it perfectly:
Ruby's Exception vs StandardError: What's the difference?
Why you shouldn't rescue Exception
The problem with rescuing Exception
is that it actually rescues every exception that inherits from
Exception. Which is....all of them!
That's a problem because there are some exceptions that are used
internally by Ruby. They don't have anything to do with your app, and
swallowing them will cause bad things to happen.
Here are a few of the big ones:
SignalException::Interrupt - If you rescue this, you can't exit your
app by hitting control-c.
ScriptError::SyntaxError - Swallowing syntax errors means that things
like puts("Forgot something) will fail silently.
NoMemoryError - Wanna know what happens when your program keeps
running after it uses up all the RAM? Me neither.
begin
do_something()
rescue Exception => e
# Don't do this. This will swallow every single exception. Nothing gets past it.
end
I'm guessing that you don't really want to swallow any of these
system-level exceptions. You only want to catch all of your
application level errors. The exceptions caused YOUR code.
Luckily, there's an easy way to to this.
Rescue StandardError Instead
All of the exceptions that you should care about inherit from StandardError. These are our old friends:
NoMethodError - raised when you try to invoke a method that doesn't exist
TypeError - caused by things like 1 + ""
RuntimeError - who could forget good old RuntimeError?
To rescue errors like these, you'll want to rescue StandardError. You COULD do it by writing something like this:
begin
do_something()
rescue StandardError => e
# Only your app's exceptions are swallowed. Things like SyntaxErrror are left alone.
end
But Ruby has made it much easier for use.
When you don't specify an exception class at all, ruby assumes you mean StandardError. So the code below is identical to the above code:
begin
do_something()
rescue => e
# This is the same as rescuing StandardError
end
This is more of an opinion oriented question. When handling exceptions in nested codes such as:
Assuming you have a class that initialize another class to run a job. The job returns a value, which is then processed by the class which initially called it.
Where would you put the exception and error logging? Would you define it on the initialization of the job class in the calling class, which will handle then exception in the job execution or on both levels ?
if the job handles exceptions then you don't need to wrap the call to the job in a try catch.
but the class that initializes and runs the job could throw exceptions, so you should handle exceptions at that level as well.
here is an example:
def some_job
begin
# a bunch of logic
rescue
# handle exception
# log it
end
end
it wouldn't make sense then to do this:
def some_manager
begin
some_job
rescue
# log
end
end
but something like this makes more sense:
def some_manager
begin
# a bunch of logic
some_job
# some more logic
rescue
# handle exception
# log
end
end
and of course you would want to catch specific exceptions.
Probably the best answer, in general, for handling Exceptions in Ruby is reading Exceptional Ruby. It may change your perspective on error handling.
Having said that, your specific case. When I hear "job" in hear "background process", so I'll base my answer on that.
Your job will want to report status while it's doing it's thing. This could be states like "in queue", "running", "finished", but it also could be more informative (user facing) information: "processing first 100 out of 1000 records".
So, if an error happens in your background process, my suggestion is two-fold:
Make sure you catch exceptions before you exit the job. Your background job processor might not like a random exception coming from your code. I, personally, like the idea of catching the exception and saving it to the database, for easy retrieval later. Then again, depending on your background job processor, maybe it handles error reporting for you. (I think reque does, for example).
On the front end, use AJAX (or something) to occasionally check in to how the job is doing. Say every 10 seconds or something. In additional to getting the status of the job, also make sure you return this additional information to the user (if appropriate).