I have an web based app that I built using Sinatra. From recent I was needing to collect data at regular intervals and store them in a database. For this I was told that I could use Resque and Clockwork gems in combine.
Every hour or so I need to do nearly 15 calculations based on the database and store the results in a database.
So this is the approach I took. I decided to make 15 classes that have the perform method ( the exact file I used for testing is below ). Then to do some thing similar to Resque.enqueue( GraphData ) for all 15 classes.
class GraphData
#queue = :graph_data
def self.init()
end
def self.perform()
File.open( '/home/ziyan/Desktop/resque.txt', 'a' ) { | file | file.write( "Resqueu - performed - #{Time.now}\n" ) }
end
end
To trigger the operation for testing purposes, I created a rake task.
desc "Start Resque Workers for Queue" # {{{
task :graph_data do |t|
ENV["QUEUE"] = "*"
ENV["VVERBOSE"] = "1"
ENV["INTERVAL"] = "5"
Resque.enqueue( GraphData )
#resque = Resque.new
#resque << AdminWorker.new
end # }}}
As you see, in the GraphData class, under self.perform method I am writing to a file.
My problem is it doesn't! Have I done some thing wrong?
rake graph_data will show no output. The web interface will show a job in the Queue.
Additional Information
I added another Rake task for running the web interface.
desc "Start Resque Web Frontend" # {{{
task :resque_web_frontend do |t|
sh "resque-web -p 8282"
puts "Resque Web Frontend is running on http://localhost:8282"
end # }}
Over their, some thing interesting is seen. As I run the rake task, under stats the pending value increases but not the processed.
Under queues -> graph_data I see some thing like this.
Class Args
GraphData []
GraphData []
GraphData []
What I ended up with:
desc "Start Resque Workers for Queue" # {{{
task :graph_data do |t|
ENV["QUEUE"] = "*"
ENV["VVERBOSE"] = "1"
ENV["INTERVAL"] = "5"
Rake::Task[ "resque:work" ].invoke
Resque.enqueue( GraphData )
#resque = Resque.new
#resque << AdminWorker.new
end # }}}
Is this in development or production? From what you describe, it looks like you are not lauching the Resque process. How are you running your app? I'm more familiar with using Resque with Rails, but you should have to run something like:
rake resque:work QUEUE='*'
To have the resque worker start.
Related
Want to send emails after every 2 minute to the email addresses from excelsheet.
I tried using sidekiq and delayed_job but emails are shooting after a delay but at same time.
Tried delay, delay_for and some methods but not helping
worker file
class MarketingEmailsWorker
include Sidekiq::Worker
def perform(*args)
EmailList.read_file(args)
end
end
EmailList.rb model
def self.read_file(record)
list = EmailList.find(record).last
spreadsheet = Roo::Spreadsheet.open(list.file.path, extension: :xlsx)
header = spreadsheet.row(1)
(2..spreadsheet.last_row).each do |i|
row = Hash[[header, spreadsheet.row(i)].transpose]
email = row["Email"]
if email.present?
geography= row["Geography"].to_s
lname = row["Name"]
designation = row["Designation"]
Notifier.send_template_mail(geography,email,lname,designation,list.emails_template).deliver_now
end
end
end
Codewise there is not really a viable option just something weird like
def perform(*args)
while true do
EmailList.read_file(args)
sleep(120) # 2 minutes in seconds
end
end
But I don't recommend it for any production system. Since you can't really control that worker.
Better way
of solving this would be using some scheduler
There you can setup a yml with cron like sidekiq workers.
There are plenty of options to get your workers scheduled e.g. cron: '*/2 * * * *' or every: '2m'.
There is also a scheduler option for delayed_job with a solid documentation.
Take in to account that as soon your worker takes longer than 2 minutes to process you will pile up workers in your queue.
RSpec 3 and sidekiq 3.2.1. And I have setup sidekiq and rspec-sidekiq properly.
Suppose I have a worker called WeatherJob, which will change the weather status from sunny to rainy:
class WeatherJob
include Sidekiq::Worker
def perform record_id
weather = Weather.find record_id
weather.update status: 'rainy'
end
end
I use this worker like this:
WeatherJob.perform_in 15.minutes, weather.id.
In the spec, I use Timecop to mock time:
require 'rails_helper'
describe WeatherJob do
let(:weather) { create :weather, status: 'sunny' }
let(:now) { Time.current }
it 'enqueue a job' do
expect {
WeatherJob.perform_async weather.id
}.to change(WeatherJob.jobs, :size).by 1
end
context '15 mins later' do
before do
Timecop.freeze(now) do
Weather.perform_in 15.minutes, weather.id
end
end
it 'update to rainy' do
Timecop.freeze(now + 16.minutes) do
expect(weather.status).to eq 'rainy'
end
end
end
end
I could see there is job in Weather.jobs array. And time is correctly 16 mins after. But it did not execute the job? Any advices? Thanks!
Sidekiq has three testing modes: disabled, fake, and inline. The default is fake, which just pushes all jobs into a jobs array and is the behavior you are seeing. The inline mode runs the job immediately instead of enqueuing it.
To force Sidekiq to run the job inline during the test, wrap your test code in a Sidekiq::Testing.inline! block:
before do
Sidekiq::Testing.inline! do
Timecop.freeze(now) do
Weather.perform_in 15.minutes, weather.id
end
end
end
For more info on testing Sidekiq, refer to the official Testing wiki page.
Do it in two steps. First test that the job was scheduled, then execute a job inline without time delay. Here is an example
it "finishes auction (async)" do
auction = FactoryGirl.create(:auction)
auction.publish!
expect(AuctionFinishWorker).to have_enqueued_sidekiq_job(auction.id).at(auction.finishes_at)
end
it "finishes auction (sync)" do
auction = FactoryGirl.create(:auction)
auction.publish!
Sidekiq::Testing.inline! do
AuctionFinishWorker.perform_async(auction.id)
end
auction.reload
expect(auction).to be_finished
end
have_enqueued_sidekiq_job method is coming from rspec-sidekiq gem. They have active development going on at develop branch. Make sure you include it like that
gem 'rspec-sidekiq', github: "philostler/rspec-sidekiq", branch: "develop"
If you want to test the job whether it should executes 15 minutes later or not then you should split you test cases into two parts. First part, you should test that whether it inserts job which would be active in 15 minutes(using mocks). Second part, whether the job has been executed properly or not.
Weather.drain can be a hack for issue
require 'rails_helper'
describe WeatherJob do
let(:weather) { create :weather, status: 'sunny' }
let(:now) { Time.current }
it 'enqueue a job' do
expect {
WeatherJob.perform_async weather.id
}.to change(WeatherJob.jobs, :size).by 1
end
context '15 mins later' do
before do
Timecop.freeze(now) do
Weather.perform_in 15.minutes, weather.id
end
end
it 'update to rainy' do
Timecop.freeze(now + 16.minutes) do
Weather.drain
expect(weather.status).to eq 'rainy'
end
end
end
end
I'm using Celluloid::IO to do DNS query and below is my code:
require 'celluloid/io'
class MyResolver
include Celluloid::IO
def initialize
#resolver = DNSResolver.new
end
def resolve(domain)
ips = #resolver.resolve domain
#sleep 1
return {domain: domain, ip: ips}
end
end
pool = MyResolver.pool(size: 5)
domains = [
'www.google.com',
## many other record
]
futures = domains.map {|d| pool.future.resolve(d)}
futures.each do |future|
puts "#{future.value}"
end
This code works and finished in few seconds. But when I add the line sleep 1(just for learning purpose), after printing some results, the process blocked forever, which is very strange.
Thanks for any help.
sleep is an overridden keyword in Celluloid, so if you want sleep from Ruby itself, use Kernel.sleep. But that being said, as of 0.17.0-dependent branch of Celluoid::IO this error you describe does not exist ( anymore? ).
I used your reproducible failing case to test the new celluloid-pool gem being released in version 0.17.0 of Celluloid, and it is working no problem with sleep 1 as is.
Something is wrong with DNSResolver at least in that case, but you can use "thread-aware DNS resolver" Resolv from standard ruby library - no any blocks with ~25k domains array. Don't forget to catch exceptions from Resolv.
I've accidentally enqueued a bunch of jobs in Sidekiq. I do not want to wipe my entire Redis store (and reset ALL Sidekiq data and enqueued jobs to nil) but I would like to remove all enqueued jobs that can be identified by a given class. How would I do this?
These answers were helpful, but didn't answer the original question for me. It's possible those solutions are out of date.
You have to access the job's args and grab it's actual job class within the loop scope. I tried the above and it did not work as expected because job.klass does not return what you'd expect it to.
This is what it returns in the terminal currently:
queue.each do |job|
puts job.klass
end
ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper
=> nil
So my solution was to dig into the job's arguments like so:
queue = Sidekiq::Queue.new("job_queue_name")
queue.each do |job|
puts job.args.first['job_class']
job.delete if job.args.first['job_class'] == "Things::DoesThatThingJob"
end
I'm sure there's a way to write this more elegantly (maybe with a select?)
But it's readable.
Hope I was able to help others like me that were looking for something like this.
I found the Sidekiq API provides an easy way to do what I need:
queue = Sidekiq::Queue.new(queue_name)
queue.each do |job|
puts job.klass
job.delete if job.klass == job_class
end
try a method like this in a helper module, where klass is the Worker class.
def self.delete_jobs_for_worker(klass)
jobs = Sidekiq::ScheduledSet.new
jobs.select do |job|
job.klass == 'Sidekiq::Extensions::DelayedClass' &&
((job_klass, job_method, args) = YAML.load(job.args[0])) &&
job_klass == klass
end.map(&:delete)
end
I am interested in knowing what would be the best way to implement a thread based queue.
For example:
I have 10 actions which I want to execute with only 4 threads. I would like to create a queue with all the 10 actions placed linearly and start the first 4 action with 4 threads, once one of the thread is done executing, the next one will start etc - So at a time, the number of thread is either 4 or less than 4.
There is a Queue class in thread in the standard library. Using that you can do something like this:
require 'thread'
queue = Queue.new
threads = []
# add work to the queue
queue << work_unit
4.times do
threads << Thread.new do
# loop until there are no more things to do
until queue.empty?
# pop with the non-blocking flag set, this raises
# an exception if the queue is empty, in which case
# work_unit will be set to nil
work_unit = queue.pop(true) rescue nil
if work_unit
# do work
end
end
# when there is no more work, the thread will stop
end
end
# wait until all threads have completed processing
threads.each { |t| t.join }
The reason I pop with the non-blocking flag is that between the until queue.empty? and the pop another thread may have pop'ed the queue, so unless the non-blocking flag is set we could get stuck at that line forever.
If you're using MRI, the default Ruby interpreter, bear in mind that threads will not be absolutely concurrent. If your work is CPU bound you may just as well run single threaded. If you have some operation that blocks on IO you may get some parallelism, but YMMV. Alternatively, you can use an interpreter that allows full concurrency, such as jRuby or Rubinius.
There area a few gems that implement this pattern for you; parallel, peach,and mine is called threach (or jruby_threach under jruby). It's a drop-in replacement for #each but allows you to specify how many threads to run with, using a SizedQueue underneath to keep things from spiraling out of control.
So...
(1..10).threach(4) {|i| do_my_work(i) }
Not pushing my own stuff; there are plenty of good implementations out there to make things easier.
If you're using JRuby, jruby_threach is a much better implementation -- Java just offers a much richer set of threading primatives and data structures to use.
Executable descriptive example:
require 'thread'
p tasks = [
{:file => 'task1'},
{:file => 'task2'},
{:file => 'task3'},
{:file => 'task4'},
{:file => 'task5'}
]
tasks_queue = Queue.new
tasks.each {|task| tasks_queue << task}
# run workers
workers_count = 3
workers = []
workers_count.times do |n|
workers << Thread.new(n+1) do |my_n|
while (task = tasks_queue.shift(true) rescue nil) do
delay = rand(0)
sleep delay
task[:result] = "done by worker ##{my_n} (in #{delay})"
p task
end
end
end
# wait for all threads
workers.each(&:join)
# output results
puts "all done"
p tasks
You could use a thread pool. It's a fairly common pattern for this type of problem.
http://en.wikipedia.org/wiki/Thread_pool_pattern
Github seems to have a few implementations you could try out:
https://github.com/search?type=Everything&language=Ruby&q=thread+pool
Celluloid have a worker pool example that does this.
I use a gem called work_queue. Its really practic.
Example:
require 'work_queue'
wq = WorkQueue.new 4, 10
(1..10).each do |number|
wq.enqueue_b("Thread#{number}") do |thread_name|
puts "Hello from the #{thread_name}"
end
end
wq.join