I see how to view heroku logs live
heroku logs -t
and how to view the last n logs e.g.
heroku logs -n 500
would show the last 500 lines of logs. But the n can only be so large because heroku simply doesn't store them (forever). For example, I tried heroku logs -n 5000 and I get about 1500 lines of logs, but no more.
So suppose I wanted to be able to view logs from farther back, what would I have to do to make them available? Is there some heroku setting/addon I need to implement in order for them to be permanently made available? Or is it the application layer's responsibility to store logs somewhere (e.g. persist them in a database)?
Heroku logs retain the last 1500 logging records, that are max 1 week old.
There are several add-ons you can use on Heroku, however they will all have some limitations (max logs, max days) when using a Free Plan.
Papertrail is a good solution I can recommend: nice dashboard for queries and alerting, saved searches, Free plan including the last 7 days logs. When purchasing a plan (starting from 8$ p/month) the limitations are 'relaxed' and you can keep logs for up to 1 year.
Not an answer, but just adding some advice I received in other forums:
Papertrail is awesome, it's a log management app (nothing to do with the ruby gem of the same name)
Some suggest not worrying about collecting all logs, but simply using something like Sentry so the errors are reported, which saves sifting through all the logs.
Related
A Heroku log would look something like:
2021-07-28T13:26:07.189019+00:00 heroku[web.1]:hello world
Is it possible to change the time format so that my human brain doesn't need to do math and can understand it directly? Something like 1:26:07 PM 2021/07/28 or something similar would be nice.
While I'm on this topic, this may be a obviously answered question but is it possible for Heroku to not delete my logs when I close the website?
This is not configurable:
Timestamp - The date and time recorded at the time the log line was produced by the dyno or component. The timestamp is in the format specified by RFC5424, and includes microsecond precision.
However, Heroku logs aren't really meant to be your primary log store. As you have noticed, they don't store very much data, or retain it very long:
Logplex is designed for collating and routing log messages, not for storage. It retains the most recent 1,500 lines of your consolidated logs, which expire after 1 week.
For more production-ready persistence of logs, you can add one of the Heroku platform's available logging add-ons to your app. Most of these add-ons offer a free plan to get started.
Alternatively, you can implement your own log drains for full control over what happens to your logs.
Many of the third-party log storage options will let you configure how timestamps are displayed. For example, here's some documentation from Papertrail about how to use a more convenient time zone.
I suggest you start by reviewing the available logging addons. Many of them have free tiers. Then pick one or two and try them out.
I have a Heroku server with about $250.00 worth of monthly addons (due to upgrades Heroku Postgres and Heroku Redis). I'm no longer using the server for the foreseeable future, but would like to be able to boot the server back up at a later date with the same configuration.
Is there a way to temporarily halt all server functionality to prevent myself from getting billed, with the possibility of rebooting the server at a later date?
Well, you can step down the dynos to hobby-dev tier if you've less than 2 process types. Or you can simply shut them down. Just go to https://dashboard.heroku.com/, click on your app and then go to the 'resources' tab to control the dynos.
Stepping down heroku-redis should be easy too. It's anyway temporary storage, that you can restart/scale up later. Also see this
The only sticking point might be your Postgres DB. If it has more than 10,000 rows, you'll have to pay atleast $9 per month, and if you've more than 1Mn rows in the DB, you'll have to pay atleast $50 per month. Many times DBs collect a lot of logs data. You can consider cleaning and compacting the data if that's possible. Or you can take a local Database dump and decommission the DB and when you decide to start the app again upload the DB (this is a bit of an extreme step though, so be doubly sure that you've everything backup up.)
I am running a job on my Heroku app that generates about 300k lines of log within 5 minutes. I need to extract all of them into a file. How can I do this?
The Heroku UI only shows logs in real time, since the moment it was opened, and only keeps 10k lines.
I attached a LogDNA Add-on as a drain, but their export also only allows 10k lines export. To even have the option of export, I need to apply a search filter (I typed 2020 because all the lines start with a date, but still...). I can scroll through all the logs to see them, but as I scroll up the bottom gets truncated, so I can't even copy-paste them myself.
I then attached Sumo Logic as a drain, which is better, because the export limit is 100k. However I still need to filter the logs in 30s to 60s intervals and download separately. Also it exports to CSV file and in reverse order (newest first, not what I want) so I have to still work on the file after its downloaded.
Is there no option to get actual raw log files in full?
Is there no option to get actual raw log files in full?
There are no actual raw log files.
Heroku's architecture requires that logging be distributed. By default, its Logplex service aggregates log output from all services into a single stream and makes it available via heroku logs. However,
Logplex is designed for collating and routing log messages, not for storage. It retains the most recent 1,500 lines of your consolidated logs, which expire after 1 week.
For longer persistence you need something else. In addition to commercial logging services like those you mentioned, you have several options:
Log to a database instead of files. Something like Apache Cassandra might be a good fit.
Send your logs to a logging server via Syslog (my preference):
Syslog drains allow you to forward your Heroku logs to an external Syslog server for long-term archiving.
Send your logs to a custom logging process via HTTPS.
Log drains also support messaging via HTTPS. This makes it easy to write your own log-processing logic and run it on a web service (such as another Heroku app).
Speaking solely from the Sumo Logic point of view, since that’s the only one I’m familiar with here, you could do this with its Search Job API: https://help.sumologic.com/APIs/Search-Job-API/About-the-Search-Job-API
The Search Job API lets you kick off a search, poll it for status, and then when complete, page through the results (up to 1M records, I believe) and do whatever you want with them, such as dumping them into a CSV file.
But this is only available to trial and Enterprise accounts.
I just looked at Heroku’s docs and it does not look like they have a native way to retrieve more than 1500 and you do have to forward those logs via syslog to a separate server / service.
I think your best solution is going to depend, however, on your use-case, such as why specifically you need these logs in a CSV.
I can see the logs for a heroku app with heroku logs -t
Is there a way to easily see how many hits an app has received in, say, the past 24 hours? (preferably using a quick command in the CLI, but otherwise through the heroku webisite)
There are two ways I see here
the Heroku dashboard provides you with a Metrics tab, where you can see the throughput of your application.
If this is not exact enough, you can add a logging addon (logentries for example), then then analyze the router-logs there. Logentries provides you with counting, grouping, etc.
Same as a logging addon, but you also can add your own log drain and then analyze them yourself :)
Disclaimer: Please tell me if the question is too broad, and I will do my best to narrow it down.
We have an Heroku app which is running 2 web 1X dynos. This infrastructure has been running for the last 9 months.
However, in the last few weeks, we had several episodes where the app would see its response times skyrocketing for about an hour, before returning to normal without us doing anything about it.
On the pictures below, you can find an extract of Heroku Metrics during one of these "episodes", which happened yesterday afternoon.
As you can see, response time is going up and eventually, almost any request made to the server gets a timeout. During the event, it was barely possible to even load the home page of our website, hosted on this app. Most of the times, we would get the "Application Error" Heroku page.
What I see is:
The amount of requests to the server (failed or not) was not crazily high (less than 1000 every 10 minutes). For this reason, I think a DDOS attack is out of the picture.
Everything that is shown by Heroku Logs is that the failed request get a 503 (Service Unavailable) error, which would make me think about an overload.
The dynos do not seem overloaded. The memory usage is low, and the dyno load is reasonable, nothing unusual.
Heroku reported no issue during our crash event, as https://status.heroku.com/ states (last incident was on the 1st of July).
Restarting the dynos through several methods (from the interface, a command line or triggering an automatic deployment via our Gitlab repository) had no effect.
I am quite unsure as to how to interpretate these metrics, and what would be the solution to ensure this kind of episode does not happen again.
So my question is: where should I look? Is there some kind of documentation about how to investigate crashes on Heroku apps?