Is there a way how to push job back to queue from sidekiq server middleware? Or simply retry without counting it?
UDPATE: My background: I need to track status of the jobs in elasticsearch (one job follows after another one), but if elastic is not accessible, and I reschedule the same worker again, I would lose the chain (jid changes).
The easiest way would be for the job to re-schedule itself, then exit. For example:
class MyJob < ApplicationJob
queue_as :default
def perform(*args)
if ready_to_perform?
# Do stuff!
else
MyJob.perform_later(args)
end
end
end
Use with caution. You probably don't want a job to be stuck re-scheduling itself forever!
This isn't quite the same as "retrying without incrementing the retry counter" (which is a little more complicated to implement), but is sufficient for most use cases like this.
This is not a working code but something like this could help you achieve want you want.
You could define a middleware and add it to sidekiq as below
Sidekiq.configure_server do |config|
config.server_middleware do |chain|
chain.add Sidekiq::RetryMonitoringMiddleware
end
end
Now, you can define in the middleware as mentioned below:
class Sidekiq::RetryMonitoringMiddleware
def call(worker, job_params, _queue)
#calling the worker perform method to add it to the queue
worker.perform(job_params['jid'], *job_params['args']) if should_retry?(job_params)
rescue StandardError => e
Rails.logger.error e
ensure
yield
end
private
def should_retry?(job)
# If worker is having a failure flag then only it should return a response
# Need to check in which key we get a failure message
(Integer(job['failure'])) == (1 || "true")
end
end
Hope it helps!!
Related
I have the following Worker
class JobBlastingWorker
include Sidekiq::Worker
sidekiq_options queue: 'job_blasting_worker'
def perform(job_id, action=nil)
job = Job.find(job_id)
JobBlastingService.new(job).call
sidekiq_id = JobBlastingWorker.perform_in(2.minutes, job.id, 're-blast', true)
job.sidekiq_trackers.create(sidekiq_id: sidekiq_id, worker_type: 'blast_version_update')
end
end
In my rspec test, i have the following job_blasting_worker_spec.erb
require 'rails_helper'
describe JobBlastingWorker do
before(:all) do
Rails.cache.clear
end
describe 'perform' do
context 'create' do
it 'creates job schedule for next 2mins' do
#job = create(:job)
worker = JobBlastingWorker.new
expect(JobBlastingWorker).to have_enqueued_sidekiq_job(#job.id, 're-blast').in(2.minutes)
worker.perform(#job.id, 'create')
end
end
end
end
I expect this to work but i realize that the sidekiq job that should be scheduled for the next 2minutes never gets created. Hence, the test fails.
How am i able to ensure that the sidekiq job actually creates for the next 2mins and the test runs successfully?
Well...for this kind of expectation, I suggest just test the message sent to the method.
expect(JobBlastingWorker).to have_enqueued_sidekiq_job(#job.id, 're-blast')
expect(JobBlastingWorker).to receive(:perform_in).with(2.minutes, job.id, 're-blast', true).and_call_original
worker = JobBlastingWorker.new
worker.perform(#job.id, 'create')
Of course, if you dig really hard, I think you will finally find a way to find the active job object in the queue, for example, by using the Redis API directly.
And then you can further examine the object and get the time you set for the job to be performed.
But why? That's ActiveJob responsibility to make sure those jobs will be performed at the right time.
Finding this doesn't help you much, and this behavior should be already tested in RSpec its tests.
I think you don't need to worry about that unless it works incorrectly and you want to reproduce that situation and issue a bug.
On the other hand, the time you send to the method is what you should care about. You don't want someone to change it to 2.hours by accident.
So I suggest you should test the message you send to the method.
Simple stuff, i need to have a way to retry a job WITHOUT RAISING THE EXCEPTION.
I know I can use something like
def perform
if stuff_happening
perform_in(2.min)
return
end
end
and its fine, but there is one problem: retry count. potentially, with stuff_happening every time, this job will keep scheduling indefinitely? is there a way to ensure it will only be scheduled a fix number of times and then stop?
Sidenote: we use 2 spaces for indent in Ruby.
You need to pass the state (amount of retries already happened) through:
def perform(attempts_left = 10)
if stuff_happening && attempts_left > 0
perform_in(2.min, attempts_left - 1)
return
end
end
I'm writing a Telegram-Bot's server in Ruby, and I want to repeat running some code. But the problem is the code I want to repeatedly run is dynamic, how can I reschedule it?
I'm not sure I am answering to your question, but it's fairly easy to reuse a block with different schedules.
require 'rufus-scheduler'
s = Rufus::Scheduler.new
job = lambda do
puts "hello #{Time.now}"
end
s.in('1s', &job)
# later on, rescheduling...
s.in('2s', &job)
s.join # just so that the example doesn't end here
You can also use a Handler and schedule it multiple times: https://github.com/jmettraux/rufus-scheduler#scheduling-handler-classes
How can I write code to go through the Resque failure queue and selectively delete jobs? Right now I've got a handful of important failures there, interspersed between thousands of failures from a runaway job that ran repeatedly. I want to delete the ones generated by the runaway job. The only API I'm familiar with is for enqueuing jobs. (I'll continue RTFMing, but I'm in a bit of a hurry.)
I neded up doing it like this:
# loop over all failure indices, instantiating as needed
(Resque::Failure.count-1).downto(0).each do |error_index_number|
failure = Resque::Failure.all(error_index_number)
# here :failure is the hash that has all the data about the failed job, perform any check you need here
if failure["error"][/regex_identifying_runaway_job/].present?
Resque::Failure.remove(error_index_number)
# or
# Resque::Failure.requeue(error_index_number)
end
As #Winfield mentioned, having a look at Resque's failure backend is useful.
You can manually modify the Failure queue the way you're asking, but it might be better to write a custom Failure handler that delete/re-enqueues jobs as they fail.
You can find the base failure backend here and an implementation that logs failed jobs to the Hoptoad exception tracking service here.
For example:
module Resque
module Failure
class RemoveRunaways < Base
def save
i=0
while job = Resque::Failure.all(i)
# Selectively remove all MyRunawayJobs from failure queue whenever they fail
if job.fetch('payload').fetch('class') == 'MyRunawayJob'
remove(i)
else
i = i + 1
end
end
end
end
end
end
EDIT: Forgot to mention how to specify this backend to handle Failures.
In your Resque initializer (eg: config/initializers/resque.rb):
# Use Resque Multi failure handler: standard handler and your custom handler
Resque::Failure::Multiple.classes = [Resque::Failure::Redis, Resque::Failure::RemoveRunaways]
Resque::Failure.backend = Resque::Failure::Multiple
Remove with bool function example
I used a higher order function approach, that evaluates a failure to remove
def remove_failures(should_remove_failure_func)
(Resque::Failure.count-1).downto(0).each do |i|
failure = Resque::Failure.all(i)
Resque::Failure.remove(i) if should_remove_failure_func.call(failure)
end
end
def remove_failed_validation_jobs
has_failed_for_validation_reason = -> (failure) do
failure["error"] == "Validation failed: Example has already been taken"
end
remove_failures(has_failed_for_validation_reason)
end
I am trying to use Ruby's daemon gem and loop the restart of a daemon that has its own loop. My code looks like this now:
require 'daemons'
while true
listener = Daemons.call(:force => true) do
users = accounts.get_updated_user_list
TweetStream::Client.new.follow(users) do |status|
puts "#{status.text}"
end
end
sleep(60)
listener.restart
end
Running this gives me the following error (after 60 seconds):
undefined method `restart' for #<Daemons::Application:0x007fc5b29f5658> (NoMethodError)
So obviously Daemons.call doesn't return a controllable daemon like I think it does. What do I need to do to set this up correctly. Is a daemon the right tool here?
I think this is what you're after, although I haven't tested it.
class RestartingUserTracker
def initialize
#client = TweetStream::Client.new
end
def handle_status(status)
# do whatever it is you're going to do with the status
end
def fetch_users
accounts.get_updated_user_list
end
def restart
#client.stop_stream
users = fetch_users
#client.follow(users) do |status|
handle_status(status)
end
end
end
EM.run do
client = RestartingUserTracker.new
client.restart
EM::PeriodicTimer.new(60) do
client.restart
end
end
Here's how it works:
TweetStream uses EventMachine internally, as a way of polling the API forever and handling the responses. I can see why you might have felt stuck, because the normal TweetStream API blocks forever and doesn't give you a way to intervene at any point. However, TweetStream does allow you to set up other things in the same event loop. In your case, a timer. I found the documentation on how to do that here: https://github.com/intridea/tweetstream#removal-of-on_interval-callback
By starting up our own EventMachine reactor, we're able to inject our own code into the reactor as well as use TweetStream. In this case, we're using a simple timer that just restarts the client every 60 seconds.
EventMachine is an implementation of something called the Reactor Pattern. If you want to fully understand and maintain this code, it would serve you well to find some resources about it and gain a full understanding. The reactor pattern is very powerful, but can be difficult to grasp at first.
However, this code should get you started. Also, I'd consider renaming the RestartingUserTracker to something more appropriate.