Is there a way to err from a Sidekiq job in a way that tells Sidekiq that "this error is fatal and unrecoverable, do not retry, send it straight to dead job queue"?
Looking at Sidekiq Error Handling documentation, it seems like it interpret all errors as transient, and will retry a job (if retry is enabled) regardless of the error type.
You should rescue those specific errors and not re-raise them.
def perform
call_something
rescue CustomException
nil
end
Edit:
Well, if you want to purposely send a message to the DLQ/DJQ, you'd need to make a method that does what #send_to_morgue does. I'm sure Mike Perham is going to come in here and yell at me for suggesting this but...
def send_to_morgue(msg)
Sidekiq.logger.info { "Adding dead #{msg['class']} job #{msg['jid']}" }
payload = Sidekiq.dump_json(msg)
now = Time.now.to_f
Sidekiq.redis do |conn|
conn.multi do
conn.zadd('dead', now, payload)
conn.zremrangebyscore('dead', '-inf', now - DeadSet.timeout)
conn.zremrangebyrank('dead', 0, -DeadSet.max_jobs)
end
end
end
The only difference you'd have to dig into what msg looks like going into that method but I suspect it's what normally hits the middleware before parse.
If found on GitHub a solution for your problem. In that post they suggested to write a custom middleware that handles the exceptions you want to prevent retries for. This is a basic example:
def call(worker, msg, queue)
begin
yield
rescue ActiveRecord::RecordNotFound => e
msg['retry'] = false
raise
end
end
You can extending that you get:
def call(worker, msg, queue)
begin
yield
rescue ActiveRecord::RecordNotFound => e
msg['retry'] = false
raise
rescue Exception => e
if worker.respond_to?(:handle_error)
worker.handle_error(e)
else
raise
end
end
end
Related
I have the following ruby code
class Gateway
...
def post
begin
...
raise ClientError if state == :open
rescue ClientError => e
Log.add("error")
raise
end
end
end
On RSpec, how can I check that when ClientError is raised Log.add is called?
I have tried different things but I always get the error raised.
Thanks
You can probably do something like this (the initialize step might need to look bit different, depending on how you need to set the state to :open):
describe 'Gateway#post' do
let(:gateway) { Gateway.new(state: :open) }
before { allow(Log).to receive(:add) }
it 'raises an excpetion' do
expect { gateway.post }.to raise_error(ClientError)
expect(Log).to have_received(:add).with('error')
end
end
Something like this should work:
describe '#post' do
context 'with state :open' do
let(:gateway) { Gateway.new(state: :open) }
it 'logs the error' do
expect(Log).to receive(:add).with('error')
gateway.post rescue nil
end
it 're-raises the error' do
expect { gateway.post }.to raise_error(ClientError)
end
end
end
In the first example, rescue nil ensures that your spec is not failing because of the raised error (it silently rescues it). The second example checks that the error is being re-raised.
With the following script
threads = [
Thread.new { Thread.current.abort_on_exception = true; raise 'err' },
Thread.new { Thread.current.abort_on_exception = true; raise 'err' },
]
begin
threads.each(&:join)
rescue RuntimeError
puts "Got Error"
end
Half the time I get the expected "Got Error" with exit 0 and the other half I get test.rb:3:in block in <main>': err (RuntimeError).
Shouldn't rescue be able to handle this? If not what might be some alternative solutions for two threads raising an error simultaneously?
I've considered not using abort_on_exception = true but the problem is that if the first thread has, say sleep(10) before the raise, the second thread, which errors immediately, won't get caught until 10 seconds is up (due to the order of the threads array).
Ruby MRI version:
ruby 2.4.0p0 (2016-12-24 revision 57164) [x86_64-darwin15]
Any ideas would be greatly appreciated. Thanks!
Update
jruby-9.1.6.0 doesn't seem to have this problem. Possibly because the inherent thread safeness of it. It always prints Got Error without any exceptions. Unfortunately, JRuby is not on option for us.
Here are a few pieces of the puzzle here.
Firstly, the program only waits for the main thread to finish:
Thread.new { Thread.current.abort_on_exception = true; raise 'Oh, no!' }
puts 'Ready or not, here I come'
The above may or may not raise the error.
Secondly, if you join on a thread, an exception raised by that threads is re-raised by the joined thread from the #join method:
gollum = Thread.new { raise 'My precious!!!' }
begin
gollum.join
rescue => e
# Prints 'My precious!!!'
puts e.message
end
Now at that point, the execution is returned to the thread that joined. It is no longer joined to the thread that caused the error or any other thread. The reason it's not joined to other threads is because you can only join one thread at the time. threads.each(&:join) actually joins you to the first, when it ends - to the second and so on:
frodo = Thread.new { raise 'Oh, no, Frodo!' }
sam = Thread.new { raise 'Oh, no, Sam!' }
begin
[frodo, sam].each(&:join)
rescue => e
puts e.message
end
puts 'This is the end, my only friend, the end.'
The above prints
Oh, no, Frodo!
This is the end, my only friend, the end.
Now lets put it together:
frodo = Thread.new { Thread.current.abort_on_exception = true; raise 'Oh, no, Frodo!' }
sam = Thread.new { Thread.current.abort_on_exception = true; raise 'Oh, no, Sam!' }
begin
[frodo, sam].each(&:join)
rescue => e
puts e.message
end
puts 'This is the end, my only friend, the end.'
Many things can happen here. What is of importance is, if we manage to join (we don't get an error before that), the rescue will catch the exception in the main thread from whichever thread manages to raise it first and then continue after the rescue. After that, the main thread (and thus the program) may or may not finish before the other thread raises its exception.
Let examine some possible outputs:
ex.rb:1:in `block in ': Oh, no, Frodo! (RuntimeError)
Frodo raised his exception before we joined.
Oh, no, Sam!
This is the end, my only friend, the end.
After we joined, Sam was the first to raise an error. After we printed the error message in the main thread, we also printed the end. Then the main thread finished, before Frodo could raise his error.
Oh, no, Frodo!ex.rb:2:in `block in ': Oh, no, Sam! (RuntimeError)
We managed to join. Frodo was the first to raise, we rescued and printed. Sam raised before we could print the end.
Oh, no, Sam!
This is the end, my only friend, the end.ex.rb:1:in `block in ': Oh, no, Frodo! (RuntimeError)
(Very rarely) We managed to get to the rescue. Sam raised an error first and we printed it from the main thread. We printed the end. Just after the print, but before the main thread is terminated, Frodo managed to jab his error as well.
As for a possible solution, you just need as many rescues as there are threads that might raise. Note that I also put the thread creation in the safeguarded block to ensure we catch potential errors before joins as well:
def execute_safely_concurrently(number_of_threads, &work)
return if number_of_threads.zero?
begin
Thread.new(&work).join
rescue => e
puts e
end
execute_safely_concurrently(number_of_threads.pred, &work)
end
execute_safely_concurrently(2) do
Thread.current.abort_on_exception = true
raise 'Handle me, bitte!'
end
After looking at #ndn's idea of wrapping each thread in it's own rescue. It looks like this solves the problem. Here is his modified example that doesn't block execution on join.
#threads = []
def execute_safely_concurrently(&work)
begin
#threads << Thread.new(&work)
rescue RuntimeError => e
puts "Child Thread Rescue: #{e}"
end
end
execute_safely_concurrently do
Thread.current.abort_on_exception = true
sleep(3)
raise 'Handle me, bitte 1!'
end
execute_safely_concurrently do
Thread.current.abort_on_exception = true
raise 'Handle me, bitte 2!'
end
begin
#threads.each(&:join)
rescue RuntimeError => e
puts "Main Thread Rescue: #{e}"
end
How to handle exceptions in concurrent-ruby thread pools (http://ruby-concurrency.github.io/concurrent-ruby/file.thread_pools.html)?
Example:
pool = Concurrent::FixedThreadPool.new(5)
pool.post do
raise 'something goes wrong'
end
# how to rescue this exception here
Update:
Here is simplified version of my code:
def process
pool = Concurrent::FixedThreadPool.new(5)
products.each do |product|
new_product = generate_new_product
pool.post do
store_in_db(new_product) # here exception is raised, e.g. connection to db failed
end
end
pool.shutdown
pool.wait_for_terminaton
end
So what I want to achive, is to stop processing (break loop) in case of any exception.
This exception is also rescued at higher level of application and there are executed some cleaning jobs (like setting state of model to failure and sending some notifications).
The following answer is from jdantonio from here https://github.com/ruby-concurrency/concurrent-ruby/issues/616
"
Most applications should not use thread pools directly. Thread pools are a low-level abstraction meant for internal use. All of the high-level abstractions in this library (Promise, Actor, etc.) all post jobs to the global thread pool and all provide exception handling. Simply pick the abstraction that best fits your use case and use it.
If you feel the need to configure your own thread pool rather than use the global thread pool, you can still use the high-level abstractions. They all support an :executor option which allows you to inject your custom thread pool. You can then use the exception handling provided by the high-level abstraction.
If you absolutely insist on posting jobs directly to a thread pool rather than using our high-level abstractions (which I strongly discourage) then just create a job wrapper. You can find examples of job wrappers in all our high-level abstractions, Rails ActiveJob, Sucker Punch, and other libraries which use our thread pools."
So how about an implementation with Promises ?
http://ruby-concurrency.github.io/concurrent-ruby/Concurrent/Promise.html
In your case it would look something like this:
promises = []
products.each do |product|
new_product = generate_new_prodcut
promises << Concurrent::Promise.execute do
store_in_db(new_product)
end
end
# .value will wait for the Thread to finish.
# The ! means, that all exceptions will be propagated to the main thread
# .zip will make one Promise which contains all other promises.
Concurrent::Promise.zip(*promises).value!
There may be a better way, but this does work. You will want to change the error handling within wait_for_pool_to_finish.
def process
pool = Concurrent::FixedThreadPool.new(10)
errors = Concurrent::Array.new
10_000.times do
pool.post do
begin
# do the work
rescue StandardError => e
errors << e
end
end
end
wait_for_pool_to_finish(pool, errors)
end
private
def wait_for_pool_to_finish(pool, errors)
pool.shutdown
until pool.shutdown?
if errors.any?
pool.kill
fail errors.first
end
sleep 1
end
pool.wait_for_termination
end
I've created an issue #634. Concurrent thread pool can support abortable worker without any problems.
require "concurrent"
Concurrent::RubyThreadPoolExecutor.class_eval do
# Inspired by "ns_kill_execution".
def ns_abort_execution aborted_worker
#pool.each do |worker|
next if worker == aborted_worker
worker.kill
end
#pool = [aborted_worker]
#ready.clear
stopped_event.set
nil
end
def abort_worker worker
synchronize do
ns_abort_execution worker
end
nil
end
def join
shutdown
# We should wait for stopped event.
# We couldn't use timeout.
stopped_event.wait nil
#pool.each do |aborted_worker|
# Rubinius could receive an error from aborted thread's "join" only.
# MRI Ruby doesn't care about "join".
# It will receive error anyway.
# We can "raise" error in aborted thread and than "join" it from this thread.
# We can "join" aborted thread from this thread and than "raise" error in aborted thread.
# The order of "raise" and "join" is not important. We will receive target error anyway.
aborted_worker.join
end
#pool.clear
nil
end
class AbortableWorker < self.const_get :Worker
def initialize pool
super
#thread.abort_on_exception = true
end
def run_task pool, task, args
begin
task.call *args
rescue StandardError => error
pool.abort_worker self
raise error
end
pool.worker_task_completed
nil
end
def join
#thread.join
nil
end
end
self.send :remove_const, :Worker
self.const_set :Worker, AbortableWorker
end
class MyError < StandardError; end
pool = Concurrent::FixedThreadPool.new 5
begin
pool.post do
sleep 1
puts "we shouldn't receive this message"
end
pool.post do
puts "raising my error"
raise MyError
end
pool.join
rescue MyError => error
puts "received my error, trace: \n#{error.backtrace.join("\n")}"
end
sleep 2
Output:
raising my error
received my error, trace:
...
This patch works fine for any version of MRI Ruby and Rubinius. JRuby is not working and I don't care. Please patch JRuby executor if you want to support it. It should be easy.
I have a sidekiq middlware which catch custom exception
require 'celluloid'
require 'sidekiq/middleware/server/retry_jobs'
module Sidekiq
class RetryMiddleware < Sidekiq::Middleware::Server::RetryJobs
def call(worker, msg, queue)
yield
rescue Sidekiq::Shutdown
# ignore, will be pushed back onto queue during hard_shutdown
raise
rescue Sidekiq::Retries::Retry => e
# force a retry (for workers that have retries disabled)
msg['retry'] = e.max_retries
attempt_retry(worker, msg, queue, e.cause)
raise e.cause
rescue Sidekiq::Retries::Fail => e
# seriously, don't retry this
raise e.cause
rescue Exception => e
# ignore, will be pushed back onto queue during hard_shutdown
raise Sidekiq::Shutdown if exception_caused_by_shutdown?(e)
raise e unless msg['retry']
attempt_retry(worker, msg, queue, e)
raise e
end
end
end
and my worker looks like below
class SomeWorker
include Sidekiq::Worker
sidekiq_options retry: false
def perform(input_data)
begin
logic to insert data into db
rescue Ione::Io::ConnectionClosedError => e
raise Sidekiq::Retries::Retry
end
end
end
When i was trying to test the SomeWorker peform method is adding the job to retry.
In Testing I am not seeing the middleware is getting called
Thanks in advance
You're making this way too hard on yourself, just call the methods.
RetryMiddleware.new.call(MyWorker.new, { ... }, 'default') do
MyWorker.new.perform(...)
end
I am trying to improve my Ruby skills by catching exceptions. I want to know if it is common to reraise the same kind of exception when you have several method calls. So, would the following code make sense? Is it ok to reraise the same kind of exception, or should I not catch it on the process method?
class Logo
def process
begin
#processed_logo = LogoProcessor::create_image(self.src)
rescue CustomException
raise CustomException
end
end
end
module LogoProcessor
def self.create_image
raise CustomException if some_condition
end
end
Sometimes we just want to know an error happened, without having to actually handle the error.
It is often the case that the one responsible for handling errors is user of the object: the caller. What if we are interested in the error, but don't want to assume that responsibility? We rescue the error, do whatever we need to do and then propagate the signal up the stack as if nothing had happened.
For example, what if we wanted to log the error message and then let the caller deal with it?
begin
this_will_fail!
rescue Failure => error
log.error error.message
raise
end
Calling raise without any arguments will raise the last error. In our case, we are re-raising error.
In the example you presented in your question, re-raising the error is simply not necessary. You could simply let it propagate up the stack naturally. The only difference in your example is you're creating a new error object and raising it instead of re-raising the last one.
This will raise the same type of error as the original, but you can customize the message.
rescue StandardError => e
raise e.class, "Message: #{e.message}"
I had the same question as in the comment thread here, i.e. What if the line before (re)raise fails?
My understanding was limited by the missing knowledge that the global variable of $! is "kinda garbage collected" // "scoped to its functional context", which the below example demonstrates:
def func
begin
raise StandardError, 'func!'
rescue StandardError => err
puts "$! = #{$!.inspect}"
end
end
begin
raise StandardError, 'oh no!'
rescue StandardError => err
func
puts "$! = #{$!.inspect}"
raise
end
The output of the above is:
$! = #<StandardError: func!>
$! = #<StandardError: oh no!>
StandardError: oh no!
from (pry):47:in `__pry__'
This behavior is different than how Python's (re)raise works.
The documentation for Exception states:
When an exception has been raised but not yet handled (in rescue,
ensure, at_exit and END blocks), two global variables are set:
$! contains the current exception.
$# contains its backtrace.
So these variables aren't true global variables, they are only defined inside the block that's handling the error.
begin
raise
rescue
p $! # StandardError
end
p $! # nil
A slightly better way to do the same thing as FreePender is to use the exception method from the Exception class, which is the ancestor class to any error classes, like StandardError, so that the method is available to any error classes.
Here the method's documentation that you can find on ApiDock:
With no argument, or if the argument is the same as the receiver, return the receiver. Otherwise, create a new exception object of the same class as the receiver, but with a message equal to string.to_str.
Now let's see how it works:
begin
this_will_fail!
rescue Failure => error
raise error.exception("Message: #{error.message}")
end
Adding to above answers here:
In some applications you may need to log the error twice.
For example exception need to be notified to monitoring tools like
Nagios/Newrelic/Cloudwatch.
At the same time you may have your own kibana backed summary logging
tool, internally for your reference.
In those cases you might want to log & handle the errors multiple times.
Example:
begin
begin
nil.to_sym
rescue => e
puts "inner block error message: #{e.message}"
puts "inner block backtrace: #{e.backtrace.join("\n")}"
raise e
end
rescue => e
puts "outer block error message: #{e.message}"
puts "outer block backtrace: #{e.backtrace.join("\n")}"
end
I am using puts here, for your the ease of verifying this code in
rails console, in actual production you may need to use rails logger