Questions
1) When running cron tasks at Heroku, does Heroku have a time limit that cron tasks should last only X minutes or can the cron task run for whatever time it needs to complete processing?
2) To optimize my application, I want to move all my processing to batch mode. So I have lots of update queries. Basically that one Heroku cron file is getting messy. What can I do to clean up the code? Should I create multiple rake tasks and invoke the task tasks from the cron file?
2.1) If you agree with that then how do I invoke rake tasks from cron task? Let's say there are 3 independent rake tasks rake accounts:billing, rake accounts:collections, rake accounts:cleanup. How do I call them in cron file?
There aren't any time limits that I know of.
Using multiple tasks instead of one big cron task is a good idea. Not only is it easier to debug and maintain when they're separate but you can also easily run them individually when needed. And well factored code is a just a good idea all by itself.
Executing a task from another task is as simple as Rake::Task[task].execute. You'd have something like this:
desc 'Heroku cron job'
task :cron => :environment do
%w{accounts:billing accounts:collections accounts:cleanup}.each do |task|
Rake::Task[task].execute
end
end
Related
I have a set of tasks in a Rakefile. I use MultiTask::invoke in order to invoke them in parallel, and it seems to work fine. These tasks eventually all share an Albacore task (an exec). This Exec task seems to have some kind of a bottleneck, whereby it runs in serial instead of in parallel. Is there some way to force Albacore to use MultiTask in order to run my Execs in parallel?
Instead of calling Task::invoke, call Task::Execute.
I've recently installed https://addons.heroku.com/scheduler in my heroku app, but I just cannot make any instruction get to work.
I think I don't know the correct syntax, for now I've tried with heroku pgbackups:capture --expire --app running-app command and selected frequency 10 mins.
It's been more than an hour and still hasn't done anything.
How can I get this to work?
Thanks
EDIT: This command is an example command, nor that I want to use that one specifically
You should be using the pgbackups addon, https://addons.heroku.com/pgbackups which is scheduled for you based on the level you pick.
Heroku Scheduler is more for if you need to run a rack task within your application codebase.
UPDATE BASED ON REVISED QUESTION:
You would write a rake task (assuming your using Rails?) which you could run locally using rake taskname and to schedule it on Heroku you would enter rake taskname as the command in the scheduler page for them to execute it.
I have a ruby on rails app (1.9.2 and 3.2) running on Heroku with Redis/Resque that requires a rake task be enqueued at regular intervals. Right now I am running 'heroku run rake update_listings' from my local machine once or twice a day....I would like to automate this. I've tried the whenever gem, but the task would not start up in the background. Heroku scheduler seems like the appropriate solution, but I am confused by the scheduler.rb file. I have:
desc "This task is called by the Heroku scheduler add-on"
task :hourly_feed => :environment do
Rake::Task[update_listings].execute
end
When I ran the :hourly_feed task from the Heroku Scheduler console and checked heroku logs, I saw several web dynos get spun up by hirefireapp, but the update_listings rake task was never invoked.
Update: I gave up on resque_scheduler. I am too green to make this work, so trying to use crontab and a sript file. Here is my update.sh script file:
Rake::Task["update_listings"].execute
I set cron using crontab-e and I have it executed every 5 minutes, but I get an error in mail logs:
Projects/livebytransit/update.sh: line 1: Rake::Task[update_listings].execute: command not found
It appears it is finding my update.sh script file and reading it, but it is not executing the code. I noticed the log entry dropped the quotes, so I also tried using single quotes in the shell script file, no change. I also tried changing the update.sh to this:
heroku run rake update_listings
error came back heroku: command not found
Personally, I used resque_scheduler, which will add jobs to the resque / redis queue using cron.
resque_schedule.yml
count_delayed_jobs_job:
cron: "0 */1 * * *"
class: Support::CountDelayedJobsResque
queue: CDJ
args:
description: "count_delayed_jobs_job, every 1hr"
alternatively, you could just chuck Rake::Task["update_listings"].execute in a shell script and use crontab to trigger the job.
It turns out the Heroku Scheduler works perfectly...I simply forgot the quotes around "updated listings".
So I have Resque and redis to go set up on Heroku and this is what my resque.rake file looks like:
require 'resque/tasks'
task "resque:setup" => :environment do
ENV['QUEUE'] = '*'
end
desc "Alias for resque:work (To run workers on Heroku)"
task "jobs:work" => "resque:work"
I ran heroku rake jobs:work and got one worker running. This worked perfectly. My background jobs were being completed.
Then I made some changes in my code, pushed to heroku, and still saw that I had one worker running. However, while jobs were being added to the queue, the worker was not receiving any jobs. So I ran heroku rake jobs:work again, it said I had two workers running, and my jobs were being completed.
My question is why did this happen? Do I need to run this rake task every time I push to heroku? Is there a way to automate this? Also, although I have two workers running, there seems to be only one that is working. Is there a way to get back to one worker?
You should use Procfile for resque jobs on heroku http://devcenter.heroku.com/articles/procfile
Keep in mind that Procfile is used on new Heroku Cedar Stack.
You only need one worker for Resque. You will need to run heroku rake jobs:work or use Resque-Scheduler (cron, or something to run that task) to to automatically run your jobs.
I'm adding Resque-Scheduler in my app on Heroku
So... I need ONE alone and distinct worker acting as the scheduler and
many doing the jobs.
This is how I've done it :
I've a distinct Heroku App which does nothing but has 1 resque-scheduler worker, running 24/7, adding Resque tasks to the Redis DB of the "distant" main App.
(I do that mapping jobs:work task to resque:scheduler or resque:work)
Is this the best way to do it on the Heroku's platform ; or am I doing it totally wrong ?
Thanks !
EDIT:
minimal app for scheduling on Heroku :
http://github.com/clmntlxndr/heroku-scheduler
UPDATE: With the new CEDAR stack on Heroku and Procfiles, it's possible to start a distinct task for each worker.
web: bundle exec rails server -p $PORT
scheduler: bundle exec rake resque:scheduler
worker: bundle exec rake jobs:work
http://devcenter.heroku.com/articles/procfile
I think I misread your question the first time. I missed the part where scheduler requires a separate rake task.
Yes, I think the only way to do it is to have two separate heroku apps, because heroku workers will only run rake jobs:work and presumably you can only map this onto one of the resque rake tasks.
You could try this:
desc "Alias for resque:work (To run workers on Heroku)"
task "jobs:work" => ["resque:work", "resque:scheduler"]
But I have sincere doubts about that actually working properly with how Heroku monitors worker processes and stuff. Also, double check the rake syntax there; it's just from memory. I know it's possible to specify multiple dependencies though.