I have 2 web dynos active on Heroku.
I'm running Unicorn and Cedar-14.
#unicorn.rb
worker_processes 3
timeout 30
#Procfile.rb
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
How can I run delayed_job using Unicorn processes? I want 2 Dynos to keep the server online but don't want to pay for an additional "worker" dyno to process some lengthy database actions.
I've seen examples for using resque but nothing concrete for Unicorn + DelayedJob.
I've been looking into the same thing lately and while I have yet to implement anything, the consensus is that the best way to accomplish this is to spin up work dynos when you need then and spin them down when you're finished.
There are a few gems that do this but from what I've seen, they all have drawbacks. I've also read that there are some services that will charge a small monthly fee to handle this which eliminated the issues the various gems had.
Related
I have deployed my node app on heroku free web dyno plan.I want to know how much free hours are remaining and how much are left so for that I am using
heroku ps -a <AppName>
After running above command I got something like this below:
As per the result everything is clear but what does Web(Free) mean written in green color. Someone please let me know any help would be appreciated.
THANKS
It means your app is running on a single web dyno and free dyno type.
Dyno configurations
Every dyno belongs to one of the three following configurations:
Web: Web dynos are dynos of the “web” process type that is defined in your Procfile. Only web dynos receive HTTP traffic from the routers.
Worker: Worker dynos can be of any process type declared in your Procfile, other than “web”. Worker dynos are typically used for background jobs, queueing systems, and timed jobs. You can have multiple kinds of worker dynos in your application. For example, one for urgent jobs and another for long-running jobs. For more information, see Worker Dynos, Background Jobs and Queueing.
One-off: One-off dynos are temporary dynos that can run detached, or with their input/output attached to your local terminal. They’re loaded with your latest release. They can be used to handle administrative tasks, such as database migrations and console sessions. They can also be used to run occasional background work, as with Heroku Scheduler. For more information, see One-Off Dynos.
Once a web or worker dyno is started, the dyno formation of your app will change (the number of running dynos of each process type) - and subject to dyno lifecycle, Heroku will continue to maintain that dyno formation until you change it. One-off dynos, on the other hand, are only expected to run a short-lived command and then exit, not affecting your dyno formation.
Dyno Types
Heroku provides a number of different dyno types each with a set of unique properties and performance characteristics. Free, Hobby, Standard and Performance dynos are available in the Common Runtime to all Heroku customers. Private Dynos only run in Private Spaces and are available in Heroku Enterprise.
I'm a little bit confused about the whole dyno business with heroku . On their site, they define a dyno as:
A dyno is a lightweight Linux container that runs a single
user-specified command. A dyno can run any command available in its
default environment (what we supply in the Cedar stack) or in your
app’s slug (a compressed and pre-packaged copy of your application and
its dependencies).
This (somewhat?) makes sense in my head, but when I think about dynos in the context of multiple dynos running for one web app, my brain gets twisted.
Lets I'm building a web app with a server which runs a very important task only once every 3 hours for all users. If I am running multiple web dynos for this site, does that mean there is a separate server instance running on each dyno? And would that very-important-task that is run every 3 hours be run on each dyno every 3 hours?
Thanks a lot for any clarification!
Each dyno is an LXC container running in one of the heroku instances.
Depending of your dyno size, there might be other containers on the same instance or not.
But the rough idea is that yes, each dyno you have running is a different instance, and a task set to run on any web dyno each 3 hours will run on all of them.
You may want to look into the heroku scheduler addon. It will run a one-off dyno at a specified interval, allowing you to run cron-like tasks.
I am using Sidekiq to process jobs. I am using Heroku basic plan which allows up to 40 connections. My understanding is that each thread can have up to 1 connection. Sidekiq has a default number of threads of 25. In my thinking I should never be getting more than 25 connections.
But I have been getting too many connections errors to redis. How would this be posible? Should I cut down the number of Sidekiq workers? Or is there something else I can do? I currently have my Procfile like this:
worker: bundle exec sidekiq
Would switching it to this fix it?
worker: bundle exec sidekiq -c 10
Is it possible Sidekiq is not closing connections properly? Also, when I get this "too many connections" error, it basically brings down the site - is there a way to let if fail gracefully which it seems like it should do.
The short answer was that heroku wasn't accurately showing the number of connections which was tripping up any debugging. And I was pretty amateurish in figuring it out since Redis has pretty much worked.
My timeout was set to zero (determined via >heroku redis:info) which basically meant that connections are held open indefinitely (the dashboard didn't show this accurately saying I was using like 10 when really having like 35).
Connecting to redis via heroku redis:cli and then running >CLIENT LIST showed the problem where there were many connections / clients probably in like TIME_WAIT state.
changing the timeout fixed this:
heroku redis:timeout --seconds 60
Honestly, calling it a connection pool is a big inaccurate.
Given that unicorn usually manages more than one Rails server process, and given that a Resque job runner probably consumes less resources than a Web request, it should be possible to run more than one resque worker on a single Heroku dyno.
Is anyone doing this successfully so far? My thoughts are, that an easy way to do so would have the Procfile runs foreman, which then runs 2 (or more) instances of the actual worker (i.e. rake resque:work)
Or is rake resque:workers up to that task? Resque itself does not recommend using that method, as this starts workers in parallel threads instead of in parallel processes.
Obviously, this makes sense only on i/o bound jobs.
One can use foreman to start multiple processes. Add foreman to your Gemfile, and then create two files:
Procfile:
worker: bundle exec foreman start -f Procfile.workers
Procfile.workers:
worker_1: QUEUE=* bundle exec rake resque:work
worker_2: QUEUE=* bundle exec rake resque:work
The same technique can be used to run a web server alongside some workers.
NOTE: while many state success using this approach, I would not suggest to use it outside of some experiments, mostly because of the risk to run into RAM limitations on small heroku instances; and once you pay for the heroku service it is probably easier to just spin up a dedicated worker machine anyways.
Based on this article, it sounds like it's possible, but the biggest gotcha is that if one of the child processes dies, Heroku won't be able to restart it.
Is it possible to run 2 different web process in one instance, like
for /url1/ some process
and
for /url2/ another
or some port manipulation?
Heroku has the concept of Dynos. They can roughly be thought of as processes.
It is possible for multiple processes to run under each Dyno. Requests are load balanced by Heroku. No special configuration is required.
For example, the Unicorn web server can be configured to spawn "worker" processes. Each "worker" serves a HTTP request. So, if you configure Unicorn to have 3 workers, 2 Dynos would give you 6 processes to serve HTTP requests.
These articles should help you understand how Heroku works:
Dynos and the Dyno Manifold
The Process Model