I'm wondering about Heroku server status and can't find any documentation about this topic.
Example:
Process exited with status 143
Can anyone explain this example? And where would I find resources for future reference?
Exit code 143 means that your process was terminated by a SIGTERM. This is generally sent when you do any commands that require your dynos to restart (config:set, restart, scale down...).
It is an idle state when it does not receive any request for a while. When it receives a request it will start again.
Daily restarts is a regularly Heroku dynos lifecycle activity:
Heroku automatic dyno restarts
It is due to the heroku app stopped by dyno.
So you have to restrat the app.
You can type heroku restart in the terminal.
Also heroku restart --app application_name
None of the answers are addressing this. It is definitely not good to be getting "process exited with status 143". It's a sign that your app is not doing things correctly.
Check out this page from the Heroku docs, specifically the sections on restarting and shutdown.
Basically, there are a number of reasons why your dyno may be restarted. Heroku does automatically restart your dyno every 24 hours, (manual restarts and deploys will reset this 24 hour period) but it can also restart your dyno for other reasons.
It's important to understand that it can be terminated at any given time, and your app needs to be designed with this in mind. For example, say you have a worker process that works some queue, popping items from the queue and doing some work on them. Wouldn't it be bad if you popped the items but then the app terminated and you couldn't do the work? Or do you have some lines of code where it might be bad if the app stopped in the middle of their execution?
Heroku does not just yank the power cord on your app; it sends a SIGTERM signal. Heroku also says (in the above link) that it's a bad idea to ignore that signal. If you're getting "process exited with status 143", it means you're not listening for that signal (for python anyway).
If you're not doing anything in your code to listen for this signal, then you're playing a dangerous game (unless it's okay for your app to be terminated at any point in its execution).
For a python app, if you don't tap into the SIGTERM signal, your app is terminated immediately (as soon as Heroku sends that signal), and you get a "process exited with status 143". Not good.
If however, you tap into that signal, then your app gets 30s to gracefully shutdown before it'll be terminated, which is ample time to finish up any work you were doing. To basically stop doing new work, and complete what you're doing if you know it'll take <30s, or put back unfinished work onto a queue, and then exit, or break whatever loop you were in. You should get "process exited with status 0". That's good.
Also, if you did tap into the signal but you don't exit in 30s, then you get an "Error R12 (Exit timeout) -> At least one process failed to exit within 30 seconds of SIGTERM", and the app is terminated with SIGKILL. You get a "process exited with status 137". Also not good.
In the above link (in the shutdown section), they give an example in ruby of how to tap into that signal. And here is an example in python.
Restart the dyno, which causes the dyno to receive SIGTERM. use this command
heroku restart worker.1
and then
heroku logs
When I check the logs:
heroku[web.1]: Idling
heroku[web.1]: State changed from up to down
heroku[web.1]: Stopping all processes with SIGTERM
heroku[web.1]: Process exited with status 143
It clearly says it's because of Idling, but how can prevent that.
When I open the web app with URL, Dyno starts the app again.
but in my case I use selenium chrome driver in the background and there is no real user.
so there should be a way to check my URL every N-minutes to prevent it from shutting down
I found a New Relic APM add-ons and also http://kaffeine.herokuapp.com/ to resolve it on this post
https://stackoverflow.com/questions/5480337/easy-way-to-prevent-heroku-idling#:~:text=You%20can%20install%20the%20free,preventing%20the%20dyno%20from%20idling.
Related
Heroku has great utilities to manage web dynos. When we do a small release we use Preboot to gracefully switch over web dynos over to the new release.
However, if you want to scale down your web dynos, there is no graceful way to do so. Ideally we can mark a web dyno so that Heroku Scheduler would stop giving it new requests for 3 minutes before sending SIGTERM.
So if i have 6 dynos running, but outside of office hours I want to ps:scale web=2, it sends SIGTERM to the other 4. If there was a straggler in slow times routed to one of those 4 (instead of the remaining 2), they'd experience a crash resulting from SIGTERM.
Is there a way to gracefully scale down web dynos?
Correctly handling SIGTERM is the recommended approach for graceful shutdown, see https://devcenter.heroku.com/articles/dynos#graceful-shutdown-with-sigterm. Please note, this the same shutdown process that occurs during preboot too. So when your process receives a SIGTERM, it should stop responding to requests, and finish responding to its current requests, after 30 seconds, if the process hasn't exited, then the process is forcefully SIGKILL'd. There is no builtin way to perform rolling scaledown, you'd have to write your own interface on the Heroku API to achieve this.
I set up a log drain for my Heroku app that drains the logs via HTTPS.
I added it to my app with this toolbelt command:
heroku drains:add http://example.com --app MY_APP
Trouble is I have a chron job that runs on a separate dyno, my worker process, and that isn't draining.
I really only want to drain the logs from the worker process.
Is there a way to add a drain to ONLY the worker process? Barring that, is there a way to make it so the worker process is included in the drain?
Thanks!
At this point in time it seems not possible to filter a drain on anything.
The drain contains ALL logging from the specified app and all dynos in that app.
the only fix right now seems to be using a filter on the receiving end.
On Heroku, I use delayed_job to run asynchronous tasks. All is well until I do a git push heroku master and then the Heroku environment kills any worker threads that are in-process.
The issue here is that those jobs never get re-queued since the delayed_job table in my db shows them as still locked and running, even though the workers that used to be servicing them are long dead.
How do I prevent this situation from occurring? I'd like Heroku to wait for all delayed jobs in progress to complete or error out before closing down, or at least terminate them and allow a new worker to be assigned to them once the server comes back up post-reboot from changes being applied by my update.
Looks like you can configure DJ to handle SIGTERM and mark the in-progress jobs as failed (so they'll be restarted again):
Use this setting to throw an exception on TERM signals by adding this in your initializer:
Delayed::Worker.raise_signal_exceptions = :term
More info in this answer:
https://stackoverflow.com/a/16811844/1715829
I have two web dynos in my heroku app, and at times get a dyno automatic restart (as per heroku policy). Is the function that was going on during the restart automatically restored in the new restarted dyno? If not, is there a way I can control this restart?
Is the function that was going on during the restart automatically restored in the new restarted dyno?
no
If not, is there a way I can control this restart?
no
What you can do, is trap the SIGTERM signal that is sent to your process 10 seconds before it is SIGKILLed. This would give you time to finish current computation, stop taking web requests, do cleanup, etc. More details on the process is in the Heroku Devcenter.
My app runs on Heroku with unicorn and uses sucker_punch to send a small quantity of emails in the background without slowing the web UI. This has been working pretty well for a few weeks.
I changed the unicorn config to the Heroku recommended config. The recommended config
includes an option for the number of unicorn processes and I upped the number of processes from 2 to 3.
Apparently that was too much. The sucker_punch jobs stopped running. I have log messages that indicate when they are queued and I have messages that indicate when they start processing. The log shows them being queued but the processing never starts.
My theory is that I exceeded memory by going from 2 to 3 unicorns.
I did not find a message anywhere indicating a problem.
Q1: should I expect to find a failure messsage somewhere? Something like "attempting to start sucker_punch -- oops, not enough memory"?
Q2: Any suggestions on how I can be notified of a failure like this in the future.
Thanks.
If you are indeed exceeding dyno memory, you should find R14 or R15 errors in your logs. See https://devcenter.heroku.com/articles/error-codes#r14-memory-quota-exceeded
A more likely problem, though, given that you haven't found these errors, is that something within the perform method of your sucker punch worker is throwing an exception. I've found sucker punch tasks to be a pain to debug because it appears the lib swallows all exceptions silently. Try instantiating your task and calling perform on it from a rails console to make sure that it behaves as you expect.
For example, you should be able to do this without causing an exception:
task = YourTask.new
task.perform :something, 55