A Heroku WORKER Dyno (running in a python environment) goes to sleep after 30 mins.
The worker executes a Celery Period task schedule to send emails to users.
Here the log showing the actual behaviour":
2019-05-29T12:49:02.273852+00:00 heroku[web.1]: Idling
2019-05-29T12:49:02.278946+00:00 heroku[web.1]: State changed from up to down
2019-05-29T12:49:02.288616+00:00 heroku[worker.1]: Idling
2019-05-29T12:49:02.306209+00:00 heroku[worker.1]: State changed from up to down
2019-05-29T12:49:03.087205+00:00 heroku[worker.1]: Stopping all processes with SIGTERM
2019-05-29T12:49:03.101608+00:00 app[worker.1]:
2019-05-29T12:49:03.101688+00:00 app[worker.1]: worker: Warm shutdown (MainProcess)
2019-05-29T12:49:03.185191+00:00 app[worker.1]: [2019-05-29 12:49:03,184: INFO/MainProcess] beat: Shutting down...
2019-05-29T12:49:03.297659+00:00 heroku[web.1]: Stopping all processes with SIGTERM
2019-05-29T12:49:03.324661+00:00 app[web.1]: [2019-05-29 12:49:03 +0000] [4] [INFO] Handling signal: term
2019-05-29T12:49:03.326185+00:00 app[web.1]: [2019-05-29 12:49:03 +0000] [11] [INFO] Worker exiting (pid: 11)
2019-05-29T12:49:03.326993+00:00 app[web.1]: [2019-05-29 12:49:03 +0000] [10] [INFO] Worker exiting (pid: 10)
2019-05-29T12:49:03.626898+00:00 app[web.1]: [2019-05-29 12:49:03 +0000] [4] [INFO] Shutting down: Master
2019-05-29T12:49:03.758456+00:00 heroku[web.1]: Process exited with status 0
2019-05-29T12:49:04.817520+00:00 heroku[worker.1]: Process exited with status 0
As per research, I was of the understanding that the web dyno goes to sleep, but the worker dyno will not sleep.
https://devcenter.heroku.com/articles/free-dyno-hours :
Web:
If an app has a free web dyno, and that dyno receives no web traffic in a 30-minute period, it will sleep.
Warning NOTE: Worker dynos do not sleep, because they do not respond to web requests. Be mindful of this as they may run 24/7 and consume from your pool of hours.
I was expecting to have web on idle (av. 250 hours) + full 24/7 of the worker running Celery beat (750 hours) , to stay mainly withing the plan.
So far I have activated: https://uptimerobot.com/ with an period 5 minutes ping to the web dyno keep it running, but hours won't suffice for a full month. With this both dynos appear to stay awake.
Any ideas why this is happening?
Potential solutions?
Right now both of your programs are using hours from a shared "hour pool" on your account. If you have uptime robot pinging your web dyno every 5 minutes (which means it doesn't have time to sleep and will be up for as long as uptime robot is running) and your worker dyno is up 24/7 you simply won't be able to run them together 24/7 for free, even if you have your credit card saved. A web dyno isn't idle if you're constantly pinging it.
I'm not entirely sure where you're getting the 250 hour figure from but if you want to run these side by side you have to let one of those dynos sleep. Right now that's not happening because of uptime robot so you're probably trying to use 1460/1000 dyno hours a month and getting your dynos shut down.
Related
Heroku has great utilities to manage web dynos. When we do a small release we use Preboot to gracefully switch over web dynos over to the new release.
However, if you want to scale down your web dynos, there is no graceful way to do so. Ideally we can mark a web dyno so that Heroku Scheduler would stop giving it new requests for 3 minutes before sending SIGTERM.
So if i have 6 dynos running, but outside of office hours I want to ps:scale web=2, it sends SIGTERM to the other 4. If there was a straggler in slow times routed to one of those 4 (instead of the remaining 2), they'd experience a crash resulting from SIGTERM.
Is there a way to gracefully scale down web dynos?
Correctly handling SIGTERM is the recommended approach for graceful shutdown, see https://devcenter.heroku.com/articles/dynos#graceful-shutdown-with-sigterm. Please note, this the same shutdown process that occurs during preboot too. So when your process receives a SIGTERM, it should stop responding to requests, and finish responding to its current requests, after 30 seconds, if the process hasn't exited, then the process is forcefully SIGKILL'd. There is no builtin way to perform rolling scaledown, you'd have to write your own interface on the Heroku API to achieve this.
In the documentation it says:
The dyno manager keeps your app’s dyno formation running without any
manual intervention. It restarts crashed dynos automatically, and
moves your dynos to new locations automatically and instantly whenever
a failure in the underlying hardware occurs.
I'm wondering what "move" means exactly? I know when a variable changes heroku restarts a dyno with new settings, but "moving" implies that the dyno keeps running while its ephemeral file system is changed if there is a storage failure. But what if there is a RAM/Processor issue?
I couldn't find more info in the dyno documentation either except that:
In addition, dynos are restarted as needed for the overall health of
the system and your app. For example, the dyno manager occasionally
detects a fault in the underlying hardware and needs to move your dyno
to a new physical location. These things happen transparently and
automatically on a regular basis and are logged to your application
logs.
Does this mean that "moving" a dyno implies a restart? Will the file system be reset? Anyone has experienced these "moves"?
Not sure what's supposed to happen, but I can show what did happen to a dyno I had running using heroku run:detached. From the logs:
2017-05-02T03:55:22.317995+00:00 app[run.5217]: { callCount: 521 }
2017-05-02T03:55:47.692257+00:00 app[run.5217]: { callCount: 522 }
2017-05-02T03:56:13.157487+00:00 app[run.5217]: { callCount: 523 }
2017-05-02T03:56:29.475119+00:00 heroku[run.5217]: Relocating dyno to a new server
2017-05-02T03:56:29.483169+00:00 heroku[run.5217]: State changed from up to complete
2017-05-02T03:56:38.500261+00:00 app[run.5217]: { callCount: 524 }
2017-05-02T03:57:03.920375+00:00 app[run.5217]: { callCount: 525 }
2017-05-02T03:57:06.330797+00:00 heroku[run.5217]: Stopping all processes with SIGTERM
2017-05-02T03:57:06.852899+00:00 heroku[run.5217]: Process exited with status 14
3
The callCount output is from the task itself. I'm not sure if this implies that the dyno was working after the move and before the SIGTERM. I'm not even sure why it was shutdown; it wasn't user initiated.
'1. I currently have 1 worker dyno doing background work. I does so every day for about an hour. The rest of the time the worker dyno recognizes, that it has nothing to do and terminates. The dyno manager will respawn the dyno and after a few seconds the dyno will terminate.
It is not clear to me how this will count in computing hours, esepecially as
https://dashboard.heroku.com/account
Shows "Not available" in the current usage statistics.
The Scheduler addon doesn't seem to be the right tool for me, as they are meant for to schedule short running background tasks (according to the description)
Q1: How does a crashed worker process consume dyno hours?
'2. How is my web app being charged when idle? The documentation says "A web dyno that is idled continues to accrue usage. To stop accruing usage, you must scale the web dyno to 0". This means that an idle web app is using the same dyno hours or at a reduced rate? "continues to accrue usage" would suggest some lower rate but isn't clear out of context.
Q2: To what rate does an idling web worker consume dyno hours?
For my understanding: Heroku does not differentiate between the Dyno types when it comes to pricing. An idled Dyno will produce the same cost like a running Dyno. But Heroku says on their website: "... Workers dynos are never idled out."
A crashed Dyno will constantly try to restart with a 10 minute break (after restart fails twice in a row).
I'm wondering about Heroku server status and can't find any documentation about this topic.
Example:
Process exited with status 143
Can anyone explain this example? And where would I find resources for future reference?
Exit code 143 means that your process was terminated by a SIGTERM. This is generally sent when you do any commands that require your dynos to restart (config:set, restart, scale down...).
It is an idle state when it does not receive any request for a while. When it receives a request it will start again.
Daily restarts is a regularly Heroku dynos lifecycle activity:
Heroku automatic dyno restarts
It is due to the heroku app stopped by dyno.
So you have to restrat the app.
You can type heroku restart in the terminal.
Also heroku restart --app application_name
None of the answers are addressing this. It is definitely not good to be getting "process exited with status 143". It's a sign that your app is not doing things correctly.
Check out this page from the Heroku docs, specifically the sections on restarting and shutdown.
Basically, there are a number of reasons why your dyno may be restarted. Heroku does automatically restart your dyno every 24 hours, (manual restarts and deploys will reset this 24 hour period) but it can also restart your dyno for other reasons.
It's important to understand that it can be terminated at any given time, and your app needs to be designed with this in mind. For example, say you have a worker process that works some queue, popping items from the queue and doing some work on them. Wouldn't it be bad if you popped the items but then the app terminated and you couldn't do the work? Or do you have some lines of code where it might be bad if the app stopped in the middle of their execution?
Heroku does not just yank the power cord on your app; it sends a SIGTERM signal. Heroku also says (in the above link) that it's a bad idea to ignore that signal. If you're getting "process exited with status 143", it means you're not listening for that signal (for python anyway).
If you're not doing anything in your code to listen for this signal, then you're playing a dangerous game (unless it's okay for your app to be terminated at any point in its execution).
For a python app, if you don't tap into the SIGTERM signal, your app is terminated immediately (as soon as Heroku sends that signal), and you get a "process exited with status 143". Not good.
If however, you tap into that signal, then your app gets 30s to gracefully shutdown before it'll be terminated, which is ample time to finish up any work you were doing. To basically stop doing new work, and complete what you're doing if you know it'll take <30s, or put back unfinished work onto a queue, and then exit, or break whatever loop you were in. You should get "process exited with status 0". That's good.
Also, if you did tap into the signal but you don't exit in 30s, then you get an "Error R12 (Exit timeout) -> At least one process failed to exit within 30 seconds of SIGTERM", and the app is terminated with SIGKILL. You get a "process exited with status 137". Also not good.
In the above link (in the shutdown section), they give an example in ruby of how to tap into that signal. And here is an example in python.
Restart the dyno, which causes the dyno to receive SIGTERM. use this command
heroku restart worker.1
and then
heroku logs
When I check the logs:
heroku[web.1]: Idling
heroku[web.1]: State changed from up to down
heroku[web.1]: Stopping all processes with SIGTERM
heroku[web.1]: Process exited with status 143
It clearly says it's because of Idling, but how can prevent that.
When I open the web app with URL, Dyno starts the app again.
but in my case I use selenium chrome driver in the background and there is no real user.
so there should be a way to check my URL every N-minutes to prevent it from shutting down
I found a New Relic APM add-ons and also http://kaffeine.herokuapp.com/ to resolve it on this post
https://stackoverflow.com/questions/5480337/easy-way-to-prevent-heroku-idling#:~:text=You%20can%20install%20the%20free,preventing%20the%20dyno%20from%20idling.
I know heroku has a 30 seconds timeout limit for the dynos. But for workers, I've been executing many tasks in the past for many minutes and I had no problem. Now in a new feature of an application, I'm seeing the process in the worker is removed (Delayed Job) with a raised Timeout::Error exception if the task takes longer than 30 seconds.
I want to confirm the maximum execution time for a worker in heroku.
Thanks
There is no request timeout for workers (or dynos for that matter). Request timeouts for the users are handled by the Routing mesh.