I'm currently running ActiveJob with DelayedJob as the backend for my background jobs on Heroku with 10 worker dynos. Daily, I need to run ~2000+ jobs which require lots of interaction with Google Sheets API that can take ~60+ seconds after the worksheet has run its calculations.
As each job could potentially take more than a minute to run, I'm wondering how I can increase the efficiency of these workers? It seems to me that these 10 workers can only take on 1 task at a time. Is it possible for 1 worker to take on my jobs? Would switching my background service to Sidekiq or another service allow these workers to take on more jobs?
Any insight would be appreciated, thanks!
Related
Heroku automatically sleeps the application afrer some time in the low price tiers which is a nice thing as it reduces the cost. But in a spring boot appllication the schedulers wont run when in sleep. Does heroru offer functionality to the wake-up the app so the schedulers will run?
Pinging every 30 minutes to have the app up and running does not suffice because the app will stay up even if it is idle. I would like to somehow wake up the app before a scheduler is about to run and then let is sleep back again if it is not used until a scheduler have to run again (or someone calls the api)
Your best bet is to move your job scheduling out of the main application. That way it doesn't have to be awake for jobs to run.
One way to do that is via the Heroku Scheduler:
Scheduler is a free add-on for running jobs on your app at scheduled time intervals, much like cron in a traditional server environment
Essentially, you can add jobs by providing a command to run and a frequency. The scheduler will kick the job off at the desired time.
Timing isn't guaranteed to be perfect, and very occasionally jobs may not run at all. But this is the most affordable option, and it has worked well for me in the past. For more precise and guaranteed timing you need to run at least one dyno continuously.
My understanding is that Heroku charges Cron tasks based on the actual amount of time the task runs, using a dyno (based on: https://devcenter.heroku.com/articles/scheduler#dynohour-costs).
So, if I need a quick task to run every X minutes, I could use the cron addon to process it, instead of a worker dyno and I would be charged a fraction of the cost.
So, if everything above is true, what is the use of the Iron.io workers? They charge (about) the same as dynos from the cron jobs and they can't connect to the DB.
I have a feeling that I am missing something.
I'm under the impression that IronWorker can use the databases for Heroku. I may be mistaken about this, however.
I can't claim a great deal of experience with cron tasks on Heroku, so forgive me if I get its limitations wrong. However, I'm under the impression that scaling from one dyno to many dynos is a little bit of a process. Where IronWorker shines is really in enabling the number of worker servers to fluctuate based on demand--you have the capacity to scale at a moment's notice, but are only paying for the scale you're actually using.
Heroku's Scheduler is great. For tasks running daily, hourly or at 10 minute intervals I think it's excellent and simple.
I have a (very fast running) process I want to run every 10 seconds. I have it set up as a rake task, how should I efficiently and simply set this up on the Heroku platform while minimising my dyno usage?
Thanks
Maybe these articles give hints even though they are for Java and Python:
https://devcenter.heroku.com/articles/scheduled-jobs-custom-clock-processes-java-quartz-rabbitmq
https://devcenter.heroku.com/articles/clock-processes-python
I want to schedule jobs to happen at a specific time and date but I'm getting confused by the wide range of options for doing so.
My requirements:
These are not recurring jobs, they only need to happen once at a specified date and time
I'm the only user of the app so don't need to deal with heavy traffic
I would like to minimise the cost of running this on Heroku, i.e. not paying idle dynos
Any tips on which combinations of gems etc. to use?
Have you looked into using https://github.com/bvandenbos/resque-scheduler? You'll need the Redis To Go addon on Heroku. This will cost you $36 a month because you'll need a scheduler process running alongside your web process. However, I've done this for free. See the README here: https://github.com/austinthecoder/pinger.
Good luck!
Due to Heroku Scheduler (default Heroku add-on) doesn't allow you to schedule your job as specific time. It is best to rely on a clock process to do the job. Gem such has Clockwork could be set up please see Heroku's Clockwork guide. You need to combine Clock with a background queuer such as resque or sidekiq. I highly recommend you go for sidekiq. Please bear in mind that both resque and sidekiq requires redis which is offered by redistogo add-on and it will cost you money to run it.
I have a Sinatra app I plan on hosting on Heroku.
This application, in part, scrapes a lot of information from other pages around the net and stores the information to a database. These scrapping operations are a slow process, so I need them to run in another thread/process separate from my Sinatra app.
My plan is just to have a button for each process that I can click and the scrapping will take place in the background.
I'm unsure what's the best way to do this, complicated by what Heroku will allow.
There's a gem called hirefire specifically for that:
HireFire automatically "hires" and "fires" (aka "scales") Delayed Job
and Resque workers on Heroku. When there are no queue jobs, HireFire
will fire (shut down) all workers. If there are queued jobs, then
it'll hire (spin up) workers. The amount of workers that get hired
depends on the amount of queued jobs (the ratio can be configured by
you). HireFire is great for both high, mid and low traffic
applications. It can save you a lot of money by only hiring workers
when there are pending jobs, and then firing them again once all the
jobs have been processed. It's also capable to dramatically reducing
processing time by automatically hiring more workers when the queue
size increases.