I'm well aware why this is happening (two ruby runtimes) and that this is a common problem for people who have not read the RS FAQ or searched on SO for this before, but I've spent a couple days trying many prescribed solutions yet my rufus-scheduler continues to invoke twice.
This occurs on production only, running Rails 5.0.6, Puma server, on Heroku.
This is my scheduler.rb:
require 'rufus-scheduler'
a_scheduler = Rufus::Scheduler.new(:lockfile => ".rufus-scheduler-a.lock")
b_scheduler = Rufus::Scheduler.new(:lockfile => ".rufus-scheduler-b.lock")
unless defined?(Rails::Console) || File.split($0).last == 'rake' || !Rails.env.production?
a_scheduler.cron '0 21 * * *', overlap: false, blocking: true do
MySidekiqWorker.perform_async unless a_scheduler.down?
end
b_scheduler.every '1h', overlap: false, blocking: true do
MyOtherSidekiqWorker.perform_async unless b_scheduler.down?
end
end
I've tried lockfiles, configuring my own scheduler_lock, different parameters for .every and .cron. Moreover, it seems even though I have overlap: false and blocking: true, new instances of MyOtherSidekiqWorker will still be invoked while one is still running.
I must be missing something obvious, thanks for your help.
So, Heroku dynos not sharing the file system
The .rufus-scheduler-a.lock seen on dyno d0 is not the .rufus-scheduler-a.lock seen on dyno d1.
Your Heroku dynos do not share the same filesystem and also they do not share the same Ruby process and thus not the same rufus-scheduler instance. So overlap: false, blocking: true will not have any effect from dyno d0 to dyno d1.
You could implement a custom locking mechanism for rufus-scheduler taking inspiration from https://github.com/jmettraux/rufus-scheduler#advanced-lock-schemes (probably via the database because it's shared by your Ruby processes) but that will not help with overlap: false and blocking: true.
If you still want to have overlap: false and blocking: true, you could look at https://devcenter.heroku.com/articles/scheduled-jobs-custom-clock-processes and have the scheduling happening in a dedicated process/dyno with rufus-scheduler or Clockwork and have no need for a schedule lock.
The rest of my answer is about your code, not about the double scheduling you are experiencing.
scheduler.down?
b_scheduler.every '1h', overlap: false, blocking: true do
MyOtherSidekiqWorker.perform_async unless b_scheduler.down?
end
Why do you have this unless b_scheduler.down? if the b_scheduler is down the block will not be executed at all.
This is sufficient:
b_scheduler.every '1h', overlap: false, blocking: true do
MyOtherSidekiqWorker.perform_async
end
a_scheduler vs b_scheduler
You do not need one scheduler for each job. You can simply write:
require 'rufus-scheduler'
#scheduler = Rufus::Scheduler.new(lockfile: '.rufus-scheduler.lock')
scheduler = Rufus::Scheduler.new
unless defined?(Rails::Console) || File.split($0).last == 'rake' || !Rails.env.production?
scheduler.cron '0 21 * * *', overlap: false, blocking: true do
MySidekiqWorker.perform_async
end
scheduler.every '1h', overlap: false, blocking: true do
MyOtherSidekiqWorker.perform_async
end
end
Related
I am using sidekiq gem for queue. and I want to process my executing parallely inside the queue.
here is my code for queue
def perform(disbursement_id)
some logic...
Parallel.each(disbursement.employee_disbursements, in_threads: 2) do |employee|
amount = amount_format(employee.amount)
res = unload_company_account(cmp_acc_id, amount.to_s)
load_employee_account(employee) unless res.empty?
end
end
Now when I use Parallel.each() without threads it works good, but when i use Parallel.each(.., in_threads:3) it goes to busy state of queue.
Not sure why in_threads takes my queue to busy state. I am not able to resolve it.
Try next to make it work
Parallel.each(disbursement.employee_disbursements, in_threads: 2) do |employee|
ActiveRecord::Base.connection_pool.with_connection do
amount = amount_format(employee.amount)
res = unload_company_account(cmp_acc_id, amount.to_s)
load_employee_account(employee) unless res.empty?
end
end
Also, that issue go away when use map instead of each or pass attribute preserve_results as true or false. That is a bit mystery because:
def each(array, options={}, &block)
map(array, options.merge(:preserve_results => false), &block)
end
I am having concurrency issues between two processes after short research I have seen that temporary file is suggested solution to this problem.
So solution would be to create /tmp/global.lock and use it as global lock. Example of this I have found in this thread Mutex for Rails Processes
Make sense to me so far, but I would like to see best practice for this solution. Above explained make sense but I wonder how to check if given file is locked?
fh = File.open("/some/file/path", File::CREAT)
begin
if locked = check_file_locked?
sleep(1)
else
fh.flock(File::LOCK_EX)
# do what you need to do
end
ensure
fh.flock(File::LOCK_UN)
end
This is my understanding of solution and not sure how to implement mentioned check_file_locked?()? Also if there is best way would love to hear it.
#bjhaid's answer can cause a problem with Timeout#timeout causing an interpreter error in Rubinius. It's also unnecessarily complicated.
Here's a simpler version, using a nonblocking lock instead of timeout:
def locked? lockfile_name
f = File.open(lockfile_name, File::CREAT)
# returns false if already locked, 0 if not
ret = f.flock(File::LOCK_EX|File::LOCK_NB)
# unlocks if possible, for cleanup; this is a noop if lock not acquired
f.flock(File::LOCK_UN)
f.close
!ret # ret == false means we *couldn't* get a lock, i.e. it was locked
end
When you have an exclusive lock to a file, attempting to lock it again in ruby would wait indefinitely till the file is unlocked, so you can rely on that and set a timeout on how long ruby should would wait, this might not be the most adequate way but I would do as below:
fh = File.open("/some/file/path", File::CREAT)
fh.flock(File::LOCK_EX)
require 'timeout'
def check_file_locked?(file)
f = File.open(file, File::CREAT)
Timeout::timeout(0.001) { f.flock(File::LOCK_EX) }
f.flock(File::LOCK_UN)
false
rescue
true
ensure
f.close
end
f = File.open("/tmp/a.txt", "w+")
f.flock(File::LOCK_EX)
check_file_locked?("/tmp/a.txt") # => true
f.flock(File::LOCK_UN)
check_file_locked?("/tmp/a.txt") # => false
I am trying to run the same script in multiple daemons.
myapp.rb looks like this:
loop do
sleep 5
1 / 0 # crash it
end
my myapp_controller.rb:
require 'rubygems'
require 'daemons'
options = {
:log_output => true,
:backtrace => true,
:monitor => true,
:multiple => true,
:log_dir => '/mnt/log/',
:hard_exit => true
}
Daemons.run(File.join(File.dirname(__FILE__), 'myapp.rb'), options)
When I run ruby myapp_controller.rb start several times in a row, it creates that many daemons, as I expect. But, after a while, due to an error in myapp.rb the daemons crash and the monitor restarts just one and not all. So I end up with a single running daemon.
Why? What am I doing wrong?
I was able to reproduce the behavior. It is not anything you are doing wrong; it is the way the daemons gem behaves.
Going through the code for the daemons gem, turns out the :multiple option doesn't work well with the :monitor option.
The :monitor option works only when the daemon is run in single mode.
I have created a bug report on the daemons project page referencing this question as the source.
More info about the reproduction of the issue:
Multiple daemon processes are created when :multiple => true. Each process has its own pid file in the format of <scriptname>.rb<number>.pid.
However, only one monitor process is created (with a single <scriptname>.rb_monitor.pid file.)
Here are the list of processes started when I start the daemon process 3 times:
$ ps -fe | grep my_server
501 1758 1 0 12:25PM ?? 0:00.63 my_server.rb
501 1759 1 0 12:25PM ?? 0:00.43 my_server.rb_monitor
501 1764 1 0 12:25PM ?? 0:00.54 my_server.rb
501 1834 1 0 12:51PM ?? 0:00.31 my_server.rb
The files in the pid/log folder:
$ ls /tmp/daemons-2013-01-25/
my_server.rb.log my_server.rb1.pid my_server.rb_monitor.pid
my_server.rb0.pid my_server.rb2.pid
Until the issue is resolved, you can change your code to something like this:
#myapp_controller.rb
require 'rubygems'
require 'daemons'
number = ARGV.fetch(1)
options = {
:app_name => "daemon-#{number}" # provide app_name
:log_output => true,
:backtrace => true,
:monitor => true,
:multiple => false, # disable multiple option
:log_dir => '/mnt/log/',
:hard_exit => true
}
Daemons.run(File.join(File.dirname(__FILE__), 'myapp.rb'), options)
And then start your daemons with these commands:
ruby myapp_controller.rb start 1
ruby myapp_controller.rb start 2
...
This slightly changes your startup code, but now you will have a monitor process for each of your daemon processes.
I have a database full of URLs that I need to test HTTP response time for on a regular basis. I want to have many worker threads combing the database at all times for a URL that hasn't been tested recently, and if it finds one, test it.
Of course, this could cause multiple threads to snag the same URL from the database. I don't want this. So, I'm trying to use Mutexes to prevent this from happening. I realize there are other options at the database level (optimistic locking, pessimistic locking), but I'd at least prefer to figure out why this isn't working.
Take a look at this test code I wrote:
threads = []
mutex = Mutex.new
50.times do |i|
threads << Thread.new do
while true do
url = nil
mutex.synchronize do
url = URL.first(:locked_for_testing => false, :times_tested.lt => 150)
if url
url.locked_for_testing = true
url.save
end
end
if url
# simulate testing the url
sleep 1
url.times_tested += 1
url.save
mutex.synchronize do
url.locked_for_testing = false
url.save
end
end
end
sleep 1
end
end
threads.each { |t| t.join }
Of course there is no real URL testing here. But what should happen is at the end of the day, each URL should end up with "times_tested" equal to 150, right?
(I'm basically just trying to make sure the mutexes and worker-thread mentality are working)
But each time I run it, a few odd URLs here and there end up with times_tested equal to a much lower number, say, 37, and locked_for_testing frozen on "true"
Now as far as I can tell from my code, if any URL gets locked, it will have to unlock. So I don't understand how some URLs are ending up "frozen" like that.
There are no exceptions and I've tried adding begin/ensure but it didn't do anything.
Any ideas?
I'd use a Queue, and a master to pull what you want. if you have a single master you control what's getting accessed. This isn't perfect but it's not going to blow up because of concurrency, remember if you aren't locking the database a mutex doesn't really help you is something else accesses the db.
code completely untested
require 'thread'
queue = Queue.new
keep_running = true
# trap cntrl_c or something to reset keep_running
master = Thread.new do
while keep_running
# check if we need some work to do
if queue.size == 0
urls = URL.all(:times_tested.lt => 150)
urls.each do |u|
queue << u.id
end
# keep from spinning the queue
sleep(0.1)
end
end
end
workers = []
50.times do
workers << Thread.new do
while keep_running
# get an id
id = queue.shift
url = URL.get(id)
#do something with the url
url.save
sleep(0.1)
end
end
end
workers.each do |w|
w.join
end
I am interested in knowing what would be the best way to implement a thread based queue.
For example:
I have 10 actions which I want to execute with only 4 threads. I would like to create a queue with all the 10 actions placed linearly and start the first 4 action with 4 threads, once one of the thread is done executing, the next one will start etc - So at a time, the number of thread is either 4 or less than 4.
There is a Queue class in thread in the standard library. Using that you can do something like this:
require 'thread'
queue = Queue.new
threads = []
# add work to the queue
queue << work_unit
4.times do
threads << Thread.new do
# loop until there are no more things to do
until queue.empty?
# pop with the non-blocking flag set, this raises
# an exception if the queue is empty, in which case
# work_unit will be set to nil
work_unit = queue.pop(true) rescue nil
if work_unit
# do work
end
end
# when there is no more work, the thread will stop
end
end
# wait until all threads have completed processing
threads.each { |t| t.join }
The reason I pop with the non-blocking flag is that between the until queue.empty? and the pop another thread may have pop'ed the queue, so unless the non-blocking flag is set we could get stuck at that line forever.
If you're using MRI, the default Ruby interpreter, bear in mind that threads will not be absolutely concurrent. If your work is CPU bound you may just as well run single threaded. If you have some operation that blocks on IO you may get some parallelism, but YMMV. Alternatively, you can use an interpreter that allows full concurrency, such as jRuby or Rubinius.
There area a few gems that implement this pattern for you; parallel, peach,and mine is called threach (or jruby_threach under jruby). It's a drop-in replacement for #each but allows you to specify how many threads to run with, using a SizedQueue underneath to keep things from spiraling out of control.
So...
(1..10).threach(4) {|i| do_my_work(i) }
Not pushing my own stuff; there are plenty of good implementations out there to make things easier.
If you're using JRuby, jruby_threach is a much better implementation -- Java just offers a much richer set of threading primatives and data structures to use.
Executable descriptive example:
require 'thread'
p tasks = [
{:file => 'task1'},
{:file => 'task2'},
{:file => 'task3'},
{:file => 'task4'},
{:file => 'task5'}
]
tasks_queue = Queue.new
tasks.each {|task| tasks_queue << task}
# run workers
workers_count = 3
workers = []
workers_count.times do |n|
workers << Thread.new(n+1) do |my_n|
while (task = tasks_queue.shift(true) rescue nil) do
delay = rand(0)
sleep delay
task[:result] = "done by worker ##{my_n} (in #{delay})"
p task
end
end
end
# wait for all threads
workers.each(&:join)
# output results
puts "all done"
p tasks
You could use a thread pool. It's a fairly common pattern for this type of problem.
http://en.wikipedia.org/wiki/Thread_pool_pattern
Github seems to have a few implementations you could try out:
https://github.com/search?type=Everything&language=Ruby&q=thread+pool
Celluloid have a worker pool example that does this.
I use a gem called work_queue. Its really practic.
Example:
require 'work_queue'
wq = WorkQueue.new 4, 10
(1..10).each do |number|
wq.enqueue_b("Thread#{number}") do |thread_name|
puts "Hello from the #{thread_name}"
end
end
wq.join