I have two web dynos in my heroku app, and at times get a dyno automatic restart (as per heroku policy). Is the function that was going on during the restart automatically restored in the new restarted dyno? If not, is there a way I can control this restart?
Is the function that was going on during the restart automatically restored in the new restarted dyno?
no
If not, is there a way I can control this restart?
no
What you can do, is trap the SIGTERM signal that is sent to your process 10 seconds before it is SIGKILLed. This would give you time to finish current computation, stop taking web requests, do cleanup, etc. More details on the process is in the Heroku Devcenter.
Related
I have an application that is hosted on Heroku. The application has a process that is run on the server, that the user has the ability to start and stop. Once the user clicks 'Start', the process is supposed to stay running until the user presses 'Stop'. The functionality of the app should allow that the process can be run for extended periods of times (6 months or so) continuously.
I have deployed my app on a Heroku free dyno. While reading the Heroku documentation, I came across this page that states that Heroku Dynos are restarted automatically every 24 hours. Here is the relevant passage:
Dynos are also restarted (cycled) at least once per day to help maintain the health of applications running on Heroku. Any changes to the local filesystem will be deleted. The cycling happens once every 24 hours (plus up to 216 random minutes, to prevent every dyno for an application from restarting at the same time). Manual restarts (heroku ps:restart) and releases (deploys or changing config vars) will reset this 24 hour period. Cycling happens for all dynos, including one-off dynos, so dynos will run for a maximum of 24 hours + 216 minutes. If you have multiple dynos, they should cycle at different times based on the random 0 to 216 minutes difference. If you continually make changes to your application without a 24 hour gap, you won’t see cycling at all.
Does this mean that the user process that he/she has started will automatically be stopped when the dyno restarts? If yes, does it automatically resume the user process where it left off?
If not, I will have to find a different hosting solution since the process may need to be run 24x7x365.
Does this mean that the user process that he/she has started will automatically be stopped when the dyno restarts?
Yes it does. As it will when you change config variables, or deploy updates, or add / remove addons.
If yes, does it automatically resume the user process where it left off?
No it doesn't. Any safe-and-resume behaviour you have to implement in your application. When stopping processes, Heroku will send them a SIGTERM signal and give them 30 seconds to safe their work.
If not, I will have to find a different hosting solution since the process may need to be run 24x7x365.
I doubt that there is any hosting solution that will give you what you want. In a cloud environment restarts are a thing that happens all the time, at least for you updating your application, bugfixes, security fixes. Every hosting provider or platform that provides you with 24x7x365 uptime will also restart and replace your dynos all the time.
Heroku has great utilities to manage web dynos. When we do a small release we use Preboot to gracefully switch over web dynos over to the new release.
However, if you want to scale down your web dynos, there is no graceful way to do so. Ideally we can mark a web dyno so that Heroku Scheduler would stop giving it new requests for 3 minutes before sending SIGTERM.
So if i have 6 dynos running, but outside of office hours I want to ps:scale web=2, it sends SIGTERM to the other 4. If there was a straggler in slow times routed to one of those 4 (instead of the remaining 2), they'd experience a crash resulting from SIGTERM.
Is there a way to gracefully scale down web dynos?
Correctly handling SIGTERM is the recommended approach for graceful shutdown, see https://devcenter.heroku.com/articles/dynos#graceful-shutdown-with-sigterm. Please note, this the same shutdown process that occurs during preboot too. So when your process receives a SIGTERM, it should stop responding to requests, and finish responding to its current requests, after 30 seconds, if the process hasn't exited, then the process is forcefully SIGKILL'd. There is no builtin way to perform rolling scaledown, you'd have to write your own interface on the Heroku API to achieve this.
I plan to deploy a mini web app to resize photos to the heroku free tier. I read that the heroku file system is ephemeral - uploaded files get deleted when the dyno restarts. What I want to know is if I upload an image only for a short duration to change its properties and then download it, is there a chance that it will get deleted before I download it? That is, can the app get cycled when it is in use?
Regards,
Debashish
On a free tier a web dyno gets cycled on:
1) Dyno restarting - according to the documentation
Dynos are also restarted (cycled) at least once per day to help maintain the health of applications running on Heroku. Any changes to the local filesystem will be deleted. The cycling happens once every 24 hours (plus up to 216 random minutes, to prevent every dyno for an application from restarting at the same time).
Restart can happen at any time then, when occurring, also in progress web request could be terminated. After the restart is triggered, you have 30 seconds to graceful shutdown before the process gets killed
2) Dyno sleeping - according to the documentation
If an app has a free web dyno, and that dyno receives no web traffic in a 30-minute period, it will sleep
If your web request executes during the same session all the operations to upload/change/download the image, you should be guaranteed the file does not get deleted in the process. However, you can avoid these events using monitoring services such as Pingdom or New Relic that can prevent a web dyno from sleeping
I'm testing an app with a worker and a web dyno on Heroku free tier and I'd like to keep the worker alive to be able to execute background tasks while letting the web dyno idle. By default they both go idle in 30 mins even if I have things queued on the worker.
I understand there're ways to keep the web dyno alive (and with that the worker as well), and there're ways to keep the web alive while scaling down the worker. However I'd need the worker alive and the web in idle.
I tried running a recurring job on the worker which would
Restart the dyno.
Scale the dyno down and then back up.
Both approaches worked (as in they restarted and scaled the dyno correctly) but the worker dyno would still idle after 30 mins (as if it's dependent on the web dyno). Edit: yep, that's pretty much the case as explained here: https://devcenter.heroku.com/articles/free-dyno-hours#dyno-sleeping
I could do this form the outside but it seems I'd have to constantly check for the state since a new restart doesn't seem to give me 30 mins headway. I'd also have to expose the API key which I'd like to avoid.
If I've gotten you right, you're trying to stop the web dyno and leave the worker dyno alive.
You could do that by going to the Resources tab:
And then in the 'web' section:
Press the pencil, toggle it off and press 'Confirm'.
As a workaround I currently remove the web dyno and explicitly enable it when I need it. As explained here:
Worker-only Free dynos do not sleep since they do not respond to web
requests.
My workaround was to just create two apps that deploy automatically from the same repository. Then, all you would need to do is enable the worker dyno for one and the web dyno for the other.
I have a Heroku worker setup to do a long running job which iterates over long periods. However whenever I do an update & deploy of other files in the repo this worker restarts, which is annoying, any way to avoid this?
No. This behaviour is part of Heroku's Automatic Dyno Restarting.
You can't work around this. Instead, you need to build all parts of your app to be able to function properly despite the fact that all dynos will restart at least once every 24 hours or so, whether or not you deploy updates in your repo.
Most significantly, you need to build support for Graceful Shutdown into all your processes (e.g. web process and worker processes).