In the documentation it says:
The dyno manager keeps your app’s dyno formation running without any
manual intervention. It restarts crashed dynos automatically, and
moves your dynos to new locations automatically and instantly whenever
a failure in the underlying hardware occurs.
I'm wondering what "move" means exactly? I know when a variable changes heroku restarts a dyno with new settings, but "moving" implies that the dyno keeps running while its ephemeral file system is changed if there is a storage failure. But what if there is a RAM/Processor issue?
I couldn't find more info in the dyno documentation either except that:
In addition, dynos are restarted as needed for the overall health of
the system and your app. For example, the dyno manager occasionally
detects a fault in the underlying hardware and needs to move your dyno
to a new physical location. These things happen transparently and
automatically on a regular basis and are logged to your application
logs.
Does this mean that "moving" a dyno implies a restart? Will the file system be reset? Anyone has experienced these "moves"?
Not sure what's supposed to happen, but I can show what did happen to a dyno I had running using heroku run:detached. From the logs:
2017-05-02T03:55:22.317995+00:00 app[run.5217]: { callCount: 521 }
2017-05-02T03:55:47.692257+00:00 app[run.5217]: { callCount: 522 }
2017-05-02T03:56:13.157487+00:00 app[run.5217]: { callCount: 523 }
2017-05-02T03:56:29.475119+00:00 heroku[run.5217]: Relocating dyno to a new server
2017-05-02T03:56:29.483169+00:00 heroku[run.5217]: State changed from up to complete
2017-05-02T03:56:38.500261+00:00 app[run.5217]: { callCount: 524 }
2017-05-02T03:57:03.920375+00:00 app[run.5217]: { callCount: 525 }
2017-05-02T03:57:06.330797+00:00 heroku[run.5217]: Stopping all processes with SIGTERM
2017-05-02T03:57:06.852899+00:00 heroku[run.5217]: Process exited with status 14
3
The callCount output is from the task itself. I'm not sure if this implies that the dyno was working after the move and before the SIGTERM. I'm not even sure why it was shutdown; it wasn't user initiated.
Related
I have an application that is hosted on Heroku. The application has a process that is run on the server, that the user has the ability to start and stop. Once the user clicks 'Start', the process is supposed to stay running until the user presses 'Stop'. The functionality of the app should allow that the process can be run for extended periods of times (6 months or so) continuously.
I have deployed my app on a Heroku free dyno. While reading the Heroku documentation, I came across this page that states that Heroku Dynos are restarted automatically every 24 hours. Here is the relevant passage:
Dynos are also restarted (cycled) at least once per day to help maintain the health of applications running on Heroku. Any changes to the local filesystem will be deleted. The cycling happens once every 24 hours (plus up to 216 random minutes, to prevent every dyno for an application from restarting at the same time). Manual restarts (heroku ps:restart) and releases (deploys or changing config vars) will reset this 24 hour period. Cycling happens for all dynos, including one-off dynos, so dynos will run for a maximum of 24 hours + 216 minutes. If you have multiple dynos, they should cycle at different times based on the random 0 to 216 minutes difference. If you continually make changes to your application without a 24 hour gap, you won’t see cycling at all.
Does this mean that the user process that he/she has started will automatically be stopped when the dyno restarts? If yes, does it automatically resume the user process where it left off?
If not, I will have to find a different hosting solution since the process may need to be run 24x7x365.
Does this mean that the user process that he/she has started will automatically be stopped when the dyno restarts?
Yes it does. As it will when you change config variables, or deploy updates, or add / remove addons.
If yes, does it automatically resume the user process where it left off?
No it doesn't. Any safe-and-resume behaviour you have to implement in your application. When stopping processes, Heroku will send them a SIGTERM signal and give them 30 seconds to safe their work.
If not, I will have to find a different hosting solution since the process may need to be run 24x7x365.
I doubt that there is any hosting solution that will give you what you want. In a cloud environment restarts are a thing that happens all the time, at least for you updating your application, bugfixes, security fixes. Every hosting provider or platform that provides you with 24x7x365 uptime will also restart and replace your dynos all the time.
I plan to deploy a mini web app to resize photos to the heroku free tier. I read that the heroku file system is ephemeral - uploaded files get deleted when the dyno restarts. What I want to know is if I upload an image only for a short duration to change its properties and then download it, is there a chance that it will get deleted before I download it? That is, can the app get cycled when it is in use?
Regards,
Debashish
On a free tier a web dyno gets cycled on:
1) Dyno restarting - according to the documentation
Dynos are also restarted (cycled) at least once per day to help maintain the health of applications running on Heroku. Any changes to the local filesystem will be deleted. The cycling happens once every 24 hours (plus up to 216 random minutes, to prevent every dyno for an application from restarting at the same time).
Restart can happen at any time then, when occurring, also in progress web request could be terminated. After the restart is triggered, you have 30 seconds to graceful shutdown before the process gets killed
2) Dyno sleeping - according to the documentation
If an app has a free web dyno, and that dyno receives no web traffic in a 30-minute period, it will sleep
If your web request executes during the same session all the operations to upload/change/download the image, you should be guaranteed the file does not get deleted in the process. However, you can avoid these events using monitoring services such as Pingdom or New Relic that can prevent a web dyno from sleeping
I have a Hobby Dyno that hosts an application in Heroku in which users can upload images.
What I've noticed is the Dyno restarts during his cycle causing all images to be gone.
2018-07-27T16:23:09.914767+00:00 heroku[web.1]: Cycling
2018-07-27T16:23:09.915421+00:00 heroku[web.1]: State changed from up to starting
I am aware of solutions that involve a third-party storage or host the app in another platform altogheter.
I am wondering if there is a way to stop a dyno cycle and never make it restart such as is always in the up state?
Thank you.
There is no way to prevent dyno cycling. Heroku does this automatically at least once per day.
Heroku's entire design is based on The 12-Factor App, which states that your app's processes are disposable. Heroku accomplishes this (in part) with its ephemeral file system, which is why you must persist files using an external service.
I have two web dynos in my heroku app, and at times get a dyno automatic restart (as per heroku policy). Is the function that was going on during the restart automatically restored in the new restarted dyno? If not, is there a way I can control this restart?
Is the function that was going on during the restart automatically restored in the new restarted dyno?
no
If not, is there a way I can control this restart?
no
What you can do, is trap the SIGTERM signal that is sent to your process 10 seconds before it is SIGKILLed. This would give you time to finish current computation, stop taking web requests, do cleanup, etc. More details on the process is in the Heroku Devcenter.
I have an app which requires long running processes - typically over 2 hours (recording streaming media). Based on Heroku's website, my worker server running these processes will be restarted randomly, at least once per day.
Is there anyway to control/avoid these restarts, so as not to interrupt my long running processes?
Do other paas providers avoid this issue?
I don't know, How to control/avoid these restarts. I was also going through their documentation, They clearly state that "Dynos are also cycled at least once per day, in addition to being restarted as needed for the overall health of the system and your app."
I think, Dynos restart should only take placed when system behaves unexpected or Dynos are found in crashed state OR In month or week to clear cache memories.
You can try App42 PaaS which monitors your Apps continuously to make sure that they are up and running. If any kontena is found in crashed state, Health Monitor try to bring it back to working state. if unable than that particular kontena is deleted & replaced with a new one.
Disclaimer: I work for App42 PaaS.