I have a question. I am using resque on heroku. There are 5 workers and 5 queues.
Now, Can I run specific queue(lets say queue4) on specific worker(let's say worker 1).
No, you cannot. Resque requires that you set the QUEUE in an environment variable (or in the .resque file at the top of your project).
You cannot, on heroku, have different environment variables depending of a dyno.
What you can do, though, is the following, in your Procfile :
urgent: QUEUE=urgent rake environment resque:work
low: QUEUE=low rake environment resque:work
You will get two different workers and can scale them both separately depending of your needs.
They will both run similarly, but have different environment variables use different queues.
if you use delayed_job this works
in Procfile.
worker_l: bundle exec rake jobs:work QUEUE=worker_l
worker_m: bundle exec rake jobs:work QUEUE=worker_m
worker_h: bundle exec rake jobs:work QUEUE=worker_h
worker_u: bundle exec rake jobs:work QUEUE=worker_u
and then
handle_asynchronously :do_it, :queue => 'worker_h'
Related
I've seen people say in a couple of different places that clockwork should be run on its own dyno, with a procfile that might look something like this (example from heroku):
clock: bundle exec clockwork lib/clock.rb
Is there any reason not to run it on the same dyno as the workers? With a process that looks something like this:
worker: bundle exec clockwork clock.rb & bundle exec sidekiq -C config/sidekiq.yml -L log/sidekiq.log
It seems to work fine this way, but I'd like to know the underlying reason why people say not to do it this way. Any guidance would be appreciated. Thanks in advance.
I've recently starting running sidekiq and clockwork side by side,
like this:
web: bundle exec puma -C config/puma.rb
worker: bundle exec clockwork clock.rb & bundle exec sidekiq & wait -n
# ^ https://help.heroku.com/CTFS2TJK/how-do-i-run-multiple-processes-on-a-dyno
And so far not seeing any issues....
This is a $25/month saving for us on the heroku hosting.
what would happen if sidekicks dies? (#Anthony said)
Normally sidekiq doesn't die (for me at least), I think it's because
sidekiq is only managing jobs. The actual work happens in classes with
a perform() method, and when those methods "raise / fail", it doesn't
bring down the sidekiq parent with it. Sidekiq has a GUI that shows the retry queue
and a dead queue, so dead jobs would end up in one of those places.
btw Heroku has an article on running multiple jobs in a single "dyno":
https://help.heroku.com/CTFS2TJK/how-do-i-run-multiple-processes-on-a-dyno
OP said:
I've seen people say in a couple of different places
I'd be curious to read those comments though, if you have the links.
I have succesfully queued a job, which I can see in resque web.
But the job is never executeted, just pending. I have 1 worker on my heroku app. In resque stats the workers are 0.
In lib/tasks/resque.rb:
require 'resque/tasks'
task "resque:setup" => :environment do
ENV['QUEUE'] = '*'
Resque.after_fork do |job|
ActiveRecord::Base.establish_connection
end
end
desc "Alias for resque:work (To run workers on Heroku)"
task "jobs:work" => "resque:work"
You have to tell Heroku to start up your Resque workers via a Procfile. This is where you put the command you would run locally, like RAILS_ENV=Production QUEUE=* bundle exec rake jobs:work
This gives the added benefit of testing you Heroku Resque setup locally by using the same Profile to start Resque workers with Foreman.
Here is a link on the Heroku site that explains how to get that set up.
As #messick said you are require to execute resque workers registration with a Procfile, and just in case you dont really care of have set up enviroments just add these lines in your Procfile:
resque: env TERM_CHILD=1 COUNT=1 QUEUE=* bundle exec rake resque:work
worker: bundle exec rake resque:work COUNT=1 QUEUE=*
If require a scheduler just add this line
scheduler: bundle exec rake resque:scheduler
I have some miniapp that use delayed_job. On my localhost everything works fine, but when I deploy my app to Heroku and click on the link that should be executed by delayed_job, so nothing happen, the "task" is just saved into the table delayed_job.
In this article on heroku blog is written, that the task from delayed_job table is executed, when is run this command rake jobs:work.
But how can I run this command? Where should be the command placed? In the code, or from terminal console?
If you are running the Cedar stack, run the following from the terminal console:
heroku run rake jobs:work
If you are running the older stacks (Bamboo, Aspen, etc.):
heroku rake jobs:work
see: https://devcenter.heroku.com/articles/rake
According to the delayed_job documentation, you can also start a worker programmatically:
#!/usr/bin/env ruby
require File.dirname(__FILE__) + '/../config/environment'
Delayed::Worker.new.start
You should use a Procfile to scpecify the commands for your dynos.
For example you would have something like this in your Procfile:
appDir/Procfile
web: bundle exec rails server -p $PORT
worker: bundle exec rake jobs:work
To use this on your development machine, you should use Foreman, it's all explained at the docs.
https://devcenter.heroku.com/articles/procfile
In our case we're only running a delayed job once a month, so didn't want to have a worker dyno running constantly.
To solve this we queue up the job (with .delayed) and then use the Heroku platform API to spawn rake jobs:workoff in a one-off worker. The API call returns relatively quickly.
PlatformAPI.connect_oauth(ENV["YOUR_HEROKU_KEY"]).dyno.create(ENV["YOUR_HEROKU_APP_NAME"],{command: 'rake jobs:workoff'})
So I have Resque and redis to go set up on Heroku and this is what my resque.rake file looks like:
require 'resque/tasks'
task "resque:setup" => :environment do
ENV['QUEUE'] = '*'
end
desc "Alias for resque:work (To run workers on Heroku)"
task "jobs:work" => "resque:work"
I ran heroku rake jobs:work and got one worker running. This worked perfectly. My background jobs were being completed.
Then I made some changes in my code, pushed to heroku, and still saw that I had one worker running. However, while jobs were being added to the queue, the worker was not receiving any jobs. So I ran heroku rake jobs:work again, it said I had two workers running, and my jobs were being completed.
My question is why did this happen? Do I need to run this rake task every time I push to heroku? Is there a way to automate this? Also, although I have two workers running, there seems to be only one that is working. Is there a way to get back to one worker?
You should use Procfile for resque jobs on heroku http://devcenter.heroku.com/articles/procfile
Keep in mind that Procfile is used on new Heroku Cedar Stack.
You only need one worker for Resque. You will need to run heroku rake jobs:work or use Resque-Scheduler (cron, or something to run that task) to to automatically run your jobs.
I'm adding Resque-Scheduler in my app on Heroku
So... I need ONE alone and distinct worker acting as the scheduler and
many doing the jobs.
This is how I've done it :
I've a distinct Heroku App which does nothing but has 1 resque-scheduler worker, running 24/7, adding Resque tasks to the Redis DB of the "distant" main App.
(I do that mapping jobs:work task to resque:scheduler or resque:work)
Is this the best way to do it on the Heroku's platform ; or am I doing it totally wrong ?
Thanks !
EDIT:
minimal app for scheduling on Heroku :
http://github.com/clmntlxndr/heroku-scheduler
UPDATE: With the new CEDAR stack on Heroku and Procfiles, it's possible to start a distinct task for each worker.
web: bundle exec rails server -p $PORT
scheduler: bundle exec rake resque:scheduler
worker: bundle exec rake jobs:work
http://devcenter.heroku.com/articles/procfile
I think I misread your question the first time. I missed the part where scheduler requires a separate rake task.
Yes, I think the only way to do it is to have two separate heroku apps, because heroku workers will only run rake jobs:work and presumably you can only map this onto one of the resque rake tasks.
You could try this:
desc "Alias for resque:work (To run workers on Heroku)"
task "jobs:work" => ["resque:work", "resque:scheduler"]
But I have sincere doubts about that actually working properly with how Heroku monitors worker processes and stuff. Also, double check the rake syntax there; it's just from memory. I know it's possible to specify multiple dependencies though.