Want to send emails after every 2 minute to the email addresses from excelsheet.
I tried using sidekiq and delayed_job but emails are shooting after a delay but at same time.
Tried delay, delay_for and some methods but not helping
worker file
class MarketingEmailsWorker
include Sidekiq::Worker
def perform(*args)
EmailList.read_file(args)
end
end
EmailList.rb model
def self.read_file(record)
list = EmailList.find(record).last
spreadsheet = Roo::Spreadsheet.open(list.file.path, extension: :xlsx)
header = spreadsheet.row(1)
(2..spreadsheet.last_row).each do |i|
row = Hash[[header, spreadsheet.row(i)].transpose]
email = row["Email"]
if email.present?
geography= row["Geography"].to_s
lname = row["Name"]
designation = row["Designation"]
Notifier.send_template_mail(geography,email,lname,designation,list.emails_template).deliver_now
end
end
end
Codewise there is not really a viable option just something weird like
def perform(*args)
while true do
EmailList.read_file(args)
sleep(120) # 2 minutes in seconds
end
end
But I don't recommend it for any production system. Since you can't really control that worker.
Better way
of solving this would be using some scheduler
There you can setup a yml with cron like sidekiq workers.
There are plenty of options to get your workers scheduled e.g. cron: '*/2 * * * *' or every: '2m'.
There is also a scheduler option for delayed_job with a solid documentation.
Take in to account that as soon your worker takes longer than 2 minutes to process you will pile up workers in your queue.
Related
I am using sidekiq gem for queue. and I want to process my executing parallely inside the queue.
here is my code for queue
def perform(disbursement_id)
some logic...
Parallel.each(disbursement.employee_disbursements, in_threads: 2) do |employee|
amount = amount_format(employee.amount)
res = unload_company_account(cmp_acc_id, amount.to_s)
load_employee_account(employee) unless res.empty?
end
end
Now when I use Parallel.each() without threads it works good, but when i use Parallel.each(.., in_threads:3) it goes to busy state of queue.
Not sure why in_threads takes my queue to busy state. I am not able to resolve it.
Try next to make it work
Parallel.each(disbursement.employee_disbursements, in_threads: 2) do |employee|
ActiveRecord::Base.connection_pool.with_connection do
amount = amount_format(employee.amount)
res = unload_company_account(cmp_acc_id, amount.to_s)
load_employee_account(employee) unless res.empty?
end
end
Also, that issue go away when use map instead of each or pass attribute preserve_results as true or false. That is a bit mystery because:
def each(array, options={}, &block)
map(array, options.merge(:preserve_results => false), &block)
end
How could I de-dupe all Sidekiq queues, ie ensure each job in the queue has unique worker and arguments.
(This arises because, for example, an object is saved twice, triggering some new job each time; but we only want it to be processed. So I'm looking to periodically de-dupe queues.)
You can use sidekiq unique jobs gem - looks like it actually does what you need.
Added later:
Here is basic implementation of what you are asking for - it would not be fast, but should be OK for small queues. I've also met this problem when repacking JSON - in my environment it was necessary to re-encode json the same way.
#for proper json packing (I had an issue with it while testing)
require 'bigdecimal'
class BigDecimal
def as_json(options = nil) #:nodoc:
if finite?
self
else
NilClass::AS_JSON
end
end
end
Sidekiq.redis do |connection|
# getting items from redis
items_count = connection.llen('queue:background')
items = connection.lrange('queue:background', 0, 100)
# remove retrieved items
connection.lrem('queue:background', 0, 100)
# jobs are in json - decode them
items_decoded = items.map{|item| ActiveSupport::JSON.decode(item)}
# group them by class and arguments
grouped = items_decoded.group_by{|item| [item['class'], item['args']]}
# get second and so forth from each group
duplicated = grouped.values.delete_if{|mini_list| mini_list.length < 2}
for_deletion = duplicated.map{|a| a[0...-1]}.flatten
for_deletion_packed = for_deletion.map{|item| JSON.generate(item)}
# removing duplicates one by one
for_deletion_packed.each do |packed_item|
connection.lrem('queue:background', 0, packed_item)
end
end
I have an web based app that I built using Sinatra. From recent I was needing to collect data at regular intervals and store them in a database. For this I was told that I could use Resque and Clockwork gems in combine.
Every hour or so I need to do nearly 15 calculations based on the database and store the results in a database.
So this is the approach I took. I decided to make 15 classes that have the perform method ( the exact file I used for testing is below ). Then to do some thing similar to Resque.enqueue( GraphData ) for all 15 classes.
class GraphData
#queue = :graph_data
def self.init()
end
def self.perform()
File.open( '/home/ziyan/Desktop/resque.txt', 'a' ) { | file | file.write( "Resqueu - performed - #{Time.now}\n" ) }
end
end
To trigger the operation for testing purposes, I created a rake task.
desc "Start Resque Workers for Queue" # {{{
task :graph_data do |t|
ENV["QUEUE"] = "*"
ENV["VVERBOSE"] = "1"
ENV["INTERVAL"] = "5"
Resque.enqueue( GraphData )
#resque = Resque.new
#resque << AdminWorker.new
end # }}}
As you see, in the GraphData class, under self.perform method I am writing to a file.
My problem is it doesn't! Have I done some thing wrong?
rake graph_data will show no output. The web interface will show a job in the Queue.
Additional Information
I added another Rake task for running the web interface.
desc "Start Resque Web Frontend" # {{{
task :resque_web_frontend do |t|
sh "resque-web -p 8282"
puts "Resque Web Frontend is running on http://localhost:8282"
end # }}
Over their, some thing interesting is seen. As I run the rake task, under stats the pending value increases but not the processed.
Under queues -> graph_data I see some thing like this.
Class Args
GraphData []
GraphData []
GraphData []
What I ended up with:
desc "Start Resque Workers for Queue" # {{{
task :graph_data do |t|
ENV["QUEUE"] = "*"
ENV["VVERBOSE"] = "1"
ENV["INTERVAL"] = "5"
Rake::Task[ "resque:work" ].invoke
Resque.enqueue( GraphData )
#resque = Resque.new
#resque << AdminWorker.new
end # }}}
Is this in development or production? From what you describe, it looks like you are not lauching the Resque process. How are you running your app? I'm more familiar with using Resque with Rails, but you should have to run something like:
rake resque:work QUEUE='*'
To have the resque worker start.
Is there a way I can reserve multiple jobs from a beanstalkd queue at once?
I'm making requests to an external API that can return up to 10 results per query. They limit the number of requests I can make each day, so the more results I get per request the better.
I couldn't find any mention of this functionality in the documentation so I'm using this workaround. Does anyone know of a better way to achieve this? Or a more appropriate tool for the job than beanstalkd perhaps?
loop do
sleep(0.3)
while #beanstalk.tubes[example].peek(:ready)
jobs = []
catch(:done) do
10.times do |i|
if #beanstalk.tubes[example].peek(:ready) then
job = #beanstalk.tubes[example].reserve(0)
jobs << job.body
job.delete
else
throw(:done)
end
end
end
process(jobs)
end
end
You can reserve several jobs concurrently by calling reserve
several times in a row before deleting or releasing those jobs.
Based on the code sample you provided, it could look something
roughly like this:
loop do
timeout = nil
jobs = []
begin
10.times do |i|
jobs << #beanstalk.tubes[example].reserve(timeout)
timeout = 0
end
rescue Beaneater::TimedOutError
# nothing to do
end
process(jobs.map{|j| j.body})
jobs.map do |job|
job.delete
end
end
I have an application that reacts to messages sent by clients. One message is reload_credentials, that the application receives any time a new client registers. This message will then connect to a PostgreSQL database, do a query for all the credentials, and then store them in a regular Ruby hash ( client_id => client_token ).
Some other messages that the application may receive are start,stop,pause which are used to keep track of some session times. My point is that I envision the application functioning in the following way:
client sends a message
message gets queued
queue is being processed
However, for example, I don't want to block the reactor. Furthermore, let's imagine I have a reload_credentials message that's next in queue. I don't want any other message from the queue to be processed until the credentials are reloaded from the DB. Also, while I am processing a certain message ( like waiting for the credentials query to finish) , I want to allow other messages to be enqueued .
Could you please guide me towards solving such a problem? I'm thinking I may have to use em-synchrony, but I am not sure.
Use one of the Postgresql EM drivers, or EM.defer so that you won't block the reactor.
When you receive the 'reload_credentials' message just flip a flag that causes all subsequent messages to be enqueued. Once the 'reload_credentials' has finished, process all messages from the queue. After the queue is empty flip the flag that causes messages to be processed as they are received.
EM drivers for Postgresql are listed here: https://github.com/eventmachine/eventmachine/wiki/Protocol-Implementations
module Server
def post_init
#queue = []
#loading_credentials = false
end
def recieve_message(type, data)
return #queue << [type, data] if #loading_credentials || !#queue.empty?
return process_msg(type, data) unless :reload_credentials == type
#loading_credentials = true
reload_credentials do
#loading_credentials = false
process_queue
end
end
def reload_credentials(&when_done)
EM.defer( proc { query_and_load_credentials }, when_done )
end
def process_queue
while (type, data = #queue.shift)
process_msg(type, data)
end
end
# lots of other methods
end
EM.start_server(HOST, PORT, Server)
If you want all connections to queue messages whenever any connection receives a 'reload_connections' message you'll have to coordinate via the eigenclass.
The following is I presume, something like your current implementation:
class Worker
def initialize queue
#queue = queue
dequeue
end
def dequeue
#queue.pop do |item|
begin
work_on item
ensure
dequeue
end
end
end
def work_on item
case item.type
when :reload_credentials
# magic happens here
else
# more magic happens here
end
end
end
q = EM::Queue.new
workers = Array.new(10) { Worker.new q }
The problem above, if I understand you correctly, is that you don't want workers working on new jobs (jobs that have arrived earlier in the producer timeline), than any reload_credentials jobs. The following should service this (additional words of caution at the end).
class Worker
def initialize queue
#queue = queue
dequeue
end
def dequeue
#queue.pop do |item|
begin
work_on item
ensure
dequeue
end
end
end
def work_on item
case item.type
when :reload_credentials
# magic happens here
else
# more magic happens here
end
end
end
class LockingDispatcher
def initialize channel, queue
#channel = channel
#queue = queue
#backlog = []
#channel.subscribe method(:dispatch_with_locking)
#locked = false
end
def dispatch_with_locking item
if locked?
#backlog << item
else
# You probably want to move the specialization here out into a method or
# block that's passed into the constructor, to make the lockingdispatcher
# more of a generic processor
case item.type
when :reload_credentials
lock
deferrable = CredentialReloader.new(item).start
deferrable.callback { unlock }
deferrable.errback { unlock }
else
dispatch_without_locking item
end
end
end
def dispatch_without_locking item
#queue << item
end
def locked?
#locked
end
def lock
#locked = true
end
def unlock
#locked = false
bl = #backlog.dup
#backlog.clear
bl.each { |item| dispatch_with_locking item }
end
end
channel = EM::Channel.new
queue = EM::Queue.new
dispatcher = LockingDispatcher.new channel, queue
workers = Array.new(10) { Worker.new queue }
So, input to the first system comes in on q, but in this new system it comes in on channel. The queue is still used for work distribution among workers, but the queue is not populated while a refresh credentials operation is going on. Unfortunately, as I didn't take more time, I have not generalized the LockingDispatcher such that it isn't coupled with the item type and code for dispatching CredentialsReloader. I'll leave that to you.
You should note here that whilst this services what I understand of your original request, it is generally better to relax this kind of requirement. There are several outstanding problems that essentially cannot be eradicated without alterations in that requirement:
The system does not wait for executing jobs to complete before starting credentials jobs
The system will handle bursts of credentials jobs very badly - other items that might be processable, won't be.
In the case of a bug in the credentials code, the backlog could fill up ram and cause failure. A simple timeout might be enough to avoid catastrophic effects, iff the code is abortable, and subsequent messages are sufficiently processable to avoid further deadlocks.
It actually sounds like you have some notion of a userid in the system. If you think through your requirements, it's likely possible that you only need to backlog items that pertain to a userid who's credentials are in a refresh state. This is a different problem, that involves a different kind of dispatching. Try a hash of locked backlogs for those users, with a callback on credential completion to drain those backlogs into the workers, or some similar arrangement.
Good luck!