https://i.stack.imgur.com/7Ar6B.png
i get a ton of these error in my logs even when my dyno is offline.
does someone have any idea of where there from ?
I beat my head against this for months without getting anywhere with Heroku support. They just kept pointing me to the L10 error docs which I'd read (and understood). The issue in my case was that there was a drain present that was NOT from an add-on. It may have been added manually by a previous engineer? Not sure. In any case, this is how I debugged and resolved it:
Look at the L10 error message to see the drain id:
2021-10-12T14:58:23.642214+00:00 heroku[logplex]: Error L10 (output buffer overflow): drain 'd.89456495-cf6c-4c1b-8865-kd0238j2098' dropped 4 messages since 2021-10-12T14:57:23.753982+00:00.
Use the heroku CLI to inspect the drains:
heroku drains --json -a [app-name]
This will list all the drains, both from add-ons and others. In my case there was an old logmatic drain (d.89456495-cf6c-4c1b-8865-kd0238j2098) in there that was the source of the errors.
Manually remove the problematic drain:
heroku drains:remove [URL] -a [app-name]
Restart the dynos? Not sure if that's necessary or if you can just wait.
Heroku log drains reference: https://devcenter.heroku.com/articles/log-drains
Like #wlabs said, it is probably a drains issue. Some people report that it could be papertrail overflowing, which is common on a free plan level.
If you don't want to remove papertrail from the app, then go into your papertrail settings and configure the log filters to cut out noise you don't care about. Then wait for the reset on the next day.
for anyone having the same issue,
I think there is no need to recreate the app, just contact the support and they will help you.
For my case the response was like that:
Thank you for contacting us. I've made some changes to the backend for
[my-app-name] and [the-stg-of-my-app] that should hopefully resolve
things. Could you please restart your applications and then let me
know if your logs work after this?
and it works!!
I finally deleted and recreated the app
Related
I see how to view heroku logs live
heroku logs -t
and how to view the last n logs e.g.
heroku logs -n 500
would show the last 500 lines of logs. But the n can only be so large because heroku simply doesn't store them (forever). For example, I tried heroku logs -n 5000 and I get about 1500 lines of logs, but no more.
So suppose I wanted to be able to view logs from farther back, what would I have to do to make them available? Is there some heroku setting/addon I need to implement in order for them to be permanently made available? Or is it the application layer's responsibility to store logs somewhere (e.g. persist them in a database)?
Heroku logs retain the last 1500 logging records, that are max 1 week old.
There are several add-ons you can use on Heroku, however they will all have some limitations (max logs, max days) when using a Free Plan.
Papertrail is a good solution I can recommend: nice dashboard for queries and alerting, saved searches, Free plan including the last 7 days logs. When purchasing a plan (starting from 8$ p/month) the limitations are 'relaxed' and you can keep logs for up to 1 year.
Not an answer, but just adding some advice I received in other forums:
Papertrail is awesome, it's a log management app (nothing to do with the ruby gem of the same name)
Some suggest not worrying about collecting all logs, but simply using something like Sentry so the errors are reported, which saves sifting through all the logs.
I can see the logs for a heroku app with heroku logs -t
Is there a way to easily see how many hits an app has received in, say, the past 24 hours? (preferably using a quick command in the CLI, but otherwise through the heroku webisite)
There are two ways I see here
the Heroku dashboard provides you with a Metrics tab, where you can see the throughput of your application.
If this is not exact enough, you can add a logging addon (logentries for example), then then analyze the router-logs there. Logentries provides you with counting, grouping, etc.
Same as a logging addon, but you also can add your own log drain and then analyze them yourself :)
Has anyone come across this before? The error message doesn't provide much useful information and Googling the error messsage returns no results. My app is part of a Pipeline, I think that might have something to do with it.
Based on the time of this post, heroku is having issues with recycling dynos.
Happening again today - heroku seems to be down (4th of Feb, 2020).
For more info: https://status.heroku.com/
My app runs on Heroku with unicorn and uses sucker_punch to send a small quantity of emails in the background without slowing the web UI. This has been working pretty well for a few weeks.
I changed the unicorn config to the Heroku recommended config. The recommended config
includes an option for the number of unicorn processes and I upped the number of processes from 2 to 3.
Apparently that was too much. The sucker_punch jobs stopped running. I have log messages that indicate when they are queued and I have messages that indicate when they start processing. The log shows them being queued but the processing never starts.
My theory is that I exceeded memory by going from 2 to 3 unicorns.
I did not find a message anywhere indicating a problem.
Q1: should I expect to find a failure messsage somewhere? Something like "attempting to start sucker_punch -- oops, not enough memory"?
Q2: Any suggestions on how I can be notified of a failure like this in the future.
Thanks.
If you are indeed exceeding dyno memory, you should find R14 or R15 errors in your logs. See https://devcenter.heroku.com/articles/error-codes#r14-memory-quota-exceeded
A more likely problem, though, given that you haven't found these errors, is that something within the perform method of your sucker punch worker is throwing an exception. I've found sucker punch tasks to be a pain to debug because it appears the lib swallows all exceptions silently. Try instantiating your task and calling perform on it from a rails console to make sure that it behaves as you expect.
For example, you should be able to do this without causing an exception:
task = YourTask.new
task.perform :something, 55
I am running Rail 2.3 on bamboo-mri-1.9.2 stack on Heroku. However I am encountering some scaling issue. During peak hours of my traffic, I am seeing some Error H11 (Backlog too deep) and I can see that via New Relic monitoring that I am having quit a high back log. However, what I would like to know now, is not the length of the queue, but the number of incoming request that are rejected by Heroku (which would result in "Error H11 (Backlog too deep)" in the Heroku log).
In short, I would like to know the ratio of incoming requests being reject to the total number of incoming requests.
As Heroku said, the log file is mainly used for debugging purpose but not for analysing real live metric, is there easy/free way to do that?
Thanks!
P.S. Comment on this post How to keep log tail alive on Heroku using ssh? suggested using the papertrailapp.com website (not related to the papertrail gem apparently), but I couldn't determine whether it can give me indication of the ratio of H11 backlog too deep error.
There is no way to see the number of rejections as far as I am aware as they are turned away by the routing mesh - long before you see the request near your app.
More importantly, why are you not scaling the dynos to handle the traffic and not be turning away users with errors?