I have below piece of code :
Converter.delay.convert("some params")
Now I want this job to be run for max 1 minute. If exceeded,delayed job should raise the exception.
I tried setting up
Delayed::Worker.max_run_time = 1.minute
but it seems it sets a timeout on the worker , not on the job.
Converter class is defined in RAILS_ROOT/lib/my_converter.rb
Timeout in the job itself
require 'timeout'
class Converter
def self.convert(params)
Timeout.timeout(60) do
# your processing
end
end
end
Delayed::Worker.max_run_time=1.minute
Its the max time on each task given to worker. The execution of any task takes more than specified we get exception raised as .
execution expired (Delayed::Worker.max_run_time is only 1 minutes)
The worker continue to run and process next tasks.
Related
I have this worker process in Heroku, which does some cleaning. It runs every two hours and listens to Heroku terminate signals. It works fine, but I'm seeing 100% dyno load all the time.
My question is: How to run this kind of worker process in Heroku without 100% dyno load? The loop causes the dyno load, but what to use instead of the infinite loop?
# Scheduler here
cleanup = Rufus::Scheduler.new
cleanup.cron '* */2 * * *' do
do_some_cleaning
end
# Signal trapping
Signal.trap("TERM") {
terminate = true
shut_down
exit 0
}
# Infinite loop
while terminate == false
end
It's because you're doing an infinite loop with no sleep cycles. This means you're basically telling the CPU that every single cycle you should be immediately executing a loop condition.
This will quickly use up your CPU.
Instead, try throwing a sleep statement into your infinite loop -- this will pause execution and bring your usage down to 0% =)
while terminate == false
sleep 1
end
I should have thought about it sooner. You can actually simply join rufus-scheduler's loop:
cleanup_scheduler = Rufus::Scheduler.new
cleanup_scheduler.cron '* */2 * * *' do
do_some_cleaning
end
Signal.trap('TERM') do
shut_down
exit 0
end
cleanup_scheduler.join
That joins rufus-scheduler scheduling thread and is pseudo equivalent to:
while !terminated
sleep 0.3
trigger_schedules_if_any
end
I'm working on a production app that has multiple rails servers behind nginx loadbalancer. We are monitoring sidekiq processes with monit, and it works just fine - when sidekiq proces dies monit starts it right back.
However recently encountered a situation where one of these processes was running and visible to monit, but for some reason not visible to sidekiq. That resulted in many failed jobs and took us some time to notice that we're missing one process in sidekiq Web UI, since monit was telling us everything was fine and all processes were running. Simple restart fixed the problem.
And that bring me to my question: how do you monitor your sidekiq processes? I know i can use something like rollbar to notify me when jobs fail, but i'd like to know if there is a way to monitor process count and preferably send mail when one dies. Any suggestions?
Something that would ping sidekiq/stats and verify response.
My super simple solution to a similar problem looks like this:
# sidekiq_check.rb
namespace :sidekiq_check do
task rerun: :environment do
if Sidekiq::ProcessSet.new.size == 0
exec 'bundle exec sidekiq -d -L log/sidekiq.log -C config/sidekiq.yml -e production'
end
end
end
and then using cron/whenever
# schedule.rb
every 5.minutes do
rake 'sidekiq_check:rerun'
end
We ran into this problem where our sidekiq processes had stopped working off jobs overnight and we had no idea. It took us about 30 minutes to integrate http://deadmanssnitch.com by following these instructions.
It's not the prettiest or cheapest option but it gets the job done (integrates nicely with Pagerduty) and has saved our butt twice in the last few months.
On of our complaints with the service is the shortest grace interval we can set is 15 minutes which is too long for us. So we're evaluating similar services like Healthchecks, etc.
My approach is the following:
create a background job that does something
call the job regularly
check that the thing is being done!
so; using a cron script (or something like whenever) every 5 mins, I run :
CheckinJob.perform_later
It's now up to sidekiq (or delayed_job, or whatever active job you're using) to actually run the job.
The job just has to do something which you can check.
I used to get the job to update a record in my Status table (essentially a list of key/value records). Then I'd have a /status page which returns a :500 status code if the record hasn't been updated in the last 6 minutes.
(obviously your timing may vary)
Then I use a monitoring service to monitor the status page! (something like StatusCake)
Nowdays I have a simpler approach; I just get the background job to check in with a cron monitoring service like
IsItWorking
Dead Mans Snitch
Health Checks
The monitoring service which expects your task to check in every X mins. If your task doesn't check in - then the monitoring service will let you know.
Integration is dead simple for all the services. For Is It Working it would be:
IsItWorkingInfo::Checkin.ping(key:"CHECKIN_IDENTIFIER")
full disclosure: I wrote IsItWorking !
I use god gem to monitor my sidekiq processes. God gem makes sure that your process is always running and also can notify the process status on various channels.
ROOT = File.dirname(File.dirname(__FILE__))
God.pid_file_directory = File.join(ROOT, "tmp/pids")
God.watch do |w|
w.env = {'RAILS_ENV' => ENV['RAILS_ENV'] || 'development'}
w.name = 'sidekiq'
w.start = "bundle exec sidekiq -d -L log/sidekiq.log -C config/sidekiq.yml -e #{ENV['RAILS_ENV']}"
w.log = "#{ROOT}/log/sidekiq_god.log"
w.behavior(:clean_pid_file)
w.dir = ROOT
w.keepalive
w.restart_if do |restart|
restart.condition(:memory_usage) do |c|
c.interval = 120.seconds
c.above = 100.megabytes
c.times = [3, 5] # 3 out of 5 intervals
end
restart.condition(:cpu_usage) do |c|
c.interval = 120.seconds
c.above = 80.percent
c.times = 5
end
end
w.lifecycle do |on|
on.condition(:flapping) do |c|
c.to_state = [:start, :restart]
c.times = 5
c.within = 5.minute
c.transition = :unmonitored
c.retry_in = 10.minutes
c.retry_times = 5
c.retry_within = 1.hours
end
end
end
This is my test, but even though #timeout_exception is working during running of the code, it's empty during the test. So how can I test if this variable is set?
Then(/^the output should be '(.*)'$/) do |expectedException|
expect(#timeout_exception).to eq(expectedException)
end
This is the output of the bundle exec cucumber run.
And the output should be 'Execution Timeout Error: This deployment has taken too long to run' # features/step_definitions/my_steps.rb:309
expected: "Execution Timeout Error: This deployment has taken too long to run"
got: nil
(compared using ==)
(RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/my_steps.rb:310:in `/^the output should be '(.*)'$/'
features/timeout_lengthy_deploys.feature:25:in `And the output should be 'Execution Timeout Error: This deployment has taken too long to run''
Failing Scenarios:
cucumber features/timeout_lengthy_deploys.feature:11 # Scenario: Normal deploy that times out because it takes too long
Selenium has its own waits and if they are set less than your expected wait then you will never see your expected wait triggered. Make sense?
The following sets the max wait for page load to 5 seconds
#browser.driver.manage.timeouts.page_load = 5
Script timeout is another (generally used with Ajax)
#browser.driver.manage.timeouts.script_timeout = 5
#browser.execute_script("return jQuery.active")
Implicit Wait is the maximum wait time for Selenium to wait for an operation on an element to complete. If this triggers first then your expect will fail.
#browser.driver.manage.timeouts.implicit_wait = 5
I would suggest setting implicit_wait higher than your timeout just prior to the test and then setting it back just afterwards. BTW, if your timeout raises an exception you will need a rescue block.
I have some jobs which take very-very long time to finish. Usually 1-3 hours.
I use the gem "daemons" to daemonize the delayed workers.
If I use the rake task: "script/delayed_job stop" to stop them, then they won't stop until the
workers finish...
Is there a nice and simple solution to stop the jobs?
I have managed to do one, but any other suggestions are also welcome.
The idea is to:
trap the "TERM" signal (because daemons sends "TERM" for its spawned processes) and save the old
when the "TERM" signal comes, then finish the job (but don't yet quit)
after terminating the job setting back the old handler for "TERM"
sending the "TERM" to the process I am in (The old handler will be called)
Here is the code:
def self.prepare_to_be_stopped_by_daemons
self.old_trap = Signal.trap("TERM") do
self.should_terminate = true
end
end
def check_the_stop_signal_by_daemons
if self.class.should_terminate
Delayed::Worker.logger.debug "Asked to be TERMINATED. Calling the old trap." if Delayed::Worker.logger
self.class.restore_old_trap
Process.kill("TERM", Process.pid)
raise StopCrawlingException.new
end
end
def self.restore_old_trap
Signal.trap("TERM", old_trap)
end
How do I create a delayed job from a rake file. How should I move it into a controller and create a delayed_job that runs the task every 15 minutes.
Here is an example how my rake file:
namespace :reklamer do
task :runall => [:iqmedier, :euroads, :mikkelsen] do
# This will run after all those tasks have run
end
task :iqmedier => :environment do
require 'Mechanize'
agent = WWW::Mechanize.new
agent.get("http://www.iqmedier.dk")
end
task :euroads => :environment do
require 'Mechanize'
require 'pp'
agent = Mechanize.new { |agent|
end
task :mikkelsen => :environment do
require 'Mechanize'
agent = Mechanize.new
agent.get("http://affilate.mikkelsenmedia.dk/partnersystem/mylogins.php")
end
end
What should I change to be a delayed job (https://github.com/collectiveidea/delayed_job)?
Suggest you take a look at SimpleWorker, a cloud-based background processing / worker queue for Ruby apps. It's designed for offloading tasks, running scheduled jobs, and scaling out to handle many parallel jobs at once. It's simple, affordable, and scalable.
(Disclosure, I work for the company.)
You create your workers (in app/worker) and then in your controllers and elsewhere queue them up to run right away or schedule them for later or on a recurring basis with just a few lines of code. Here's a basic example.
worker = ReportWorker.new
worker.user_id = #current_user.id
worker.schedule(:start_at => 1.hours.since, :run_every => 900)
#Or to run once right away
#worker.queue
The ReportWorker class would contain the logic to create the report for the current user and sent it or post it needed.
DelayedJob alone will not help you since it is based around one-time jobs. You will still need something that runs on a regular basis that creates these jobs.
Assuming:
you're on Heroku and can only get a 1-hour cron
you need to run a job every 15 minutes
You can do something like this...
Make a class for your jobs:
class MechanizeJob < Struct.new(:url)
def perform
agent = Mechanize.new
agent.get(url)
end
end
Schedule the jobs from your Rakefile:
task :schedulejobs => :environment do
urls = ["http://...", "http://...", "http://..."]
urls.each do |url|
# 1 is the job priority
Delayed::Job.enqueue MechanizeJob.new(url), 1, Time.now
Delayed::Job.enqueue MechanizeJob.new(url), 1, 15.minutes.from_now
Delayed::Job.enqueue MechanizeJob.new(url), 1, 30.minutes.from_now
Delayed::Job.enqueue MechanizeJob.new(url), 1, 45.minutes.from_now
end
end
This will run a job per url every 15 minutes.