I'm running a Flask app on Heroku. I've connected the logs output to Loggly, however trying Loggly's verify step, I receive the following message:
We received your logs but they do not appear to use one of our
automated parsing formats. Please see our guide to learn more (link).
You can still do full text search and counts on these logs, but you
won't be able to use some features like filtering by field.
Why doesn't Loggly parse Heroku's logs well? What can I change so it will?
The logs appear to be quite standard for Heroku, here's an example:
2014-08-26 11:06:00 [2] [INFO] 1 workers
2014-08-26 11:05:45 [2] [INFO] 1 workers
2014-08-26 11:05:16 [2] [INFO] 1 workers
at=info method=GET path="/" host=my-app.herokuapp.com request_id=925f3605-2ac4-4182-b148-987ab7d985c4 fwd="91.271.61.73" dyno=web.1 connect=1ms service=4ms status=200 bytes=608
Process exited with status 0
2014-08-26 11:04:47 [2] [INFO] Shutting down: Master
2014-08-26 11:04:47 [2] [INFO] 1 workers
2014-08-26 11:04:47 [2] [INFO] Handling signal: term
2014-08-26 11:04:47 [7] [INFO] Worker exiting (pid: 7)
State changed from starting to up
2014-08-26 11:04:46 [2] [INFO] 1 workers
2014-08-26 11:04:46 [7] [INFO] Booting worker with pid: 7
2014-08-26 11:04:46 [2] [INFO] Listening at: http://0.0.0.0:26268 (2)
2014-08-26 11:04:46 [2] [INFO] Using worker: sync
2014-08-26 11:04:46 [2] [INFO] Starting gunicorn 19.1.0
Stopping all processes with SIGTERM
Starting process with command `gunicorn service:app --log-file=-`
2014-08-26 11:04:44 [2] [INFO] 1 workers
at=info method=GET path="/" host=my-app.herokuapp.com request_id=e152c661-b633-4870-aeba-68cea3cc8039 fwd="73.24.16.159" dyno=web.1 connect=0ms service=13ms status=200 bytes=407
Slug compilation finished
State changed from up to starting
Release v10 created by my#email.com
Deploy be32f3c by my#email.com
2014-08-26 11:04:29 [2] [INFO] 1 workers
Slug compilation started
2014-08-26 11:04:14 [2] [INFO] 1 workers
2014-08-26 11:03:59 [2] [INFO] 1 workers
Although initially I thought this was a problem on our side, Loggly's support admitted there's a problem with their parsing of Heroku logs and that they are working on it.
I'm keeping this question open until Loggly fix this problem in case others will search for it. Once Loggly fix it, I'll delete the question.
Related
This question already has answers here:
Heroku: ModuleNotFoundError :No module named 'requests'
(1 answer)
gunicorn command not found, but it's in my requirements.txt
(1 answer)
Closed 3 months ago.
When I try deploying my Flask app to Heroku, I get a ModuleNotFound Error for requests_futures.
I have included it in requirements.txt and I am still getting the issue. Heroku deploys fine without the requests_futures module. Is there anything I can do?
2022-11-07T22:26:40.279231+00:00 app[web.1]: from app.main import app
2022-11-07T22:26:40.279231+00:00 app[web.1]: File "/app/app/main.py", line 3, in <module>
2022-11-07T22:26:40.279231+00:00 app[web.1]: from .tools.web_scraping import web_scrape
2022-11-07T22:26:40.279231+00:00 app[web.1]: File "/app/app/tools/web_scraping/web_scrape.py", line 4, in <module>
2022-11-07T22:26:40.279232+00:00 app[web.1]: from requests_futures import sessions
2022-11-07T22:26:40.279232+00:00 app[web.1]: ModuleNotFoundError: No module named 'requests_futures'
2022-11-07T22:26:40.279381+00:00 app[web.1]: [2022-11-07 22:26:40 +0000] [9] [INFO] Worker exiting (pid: 9)
2022-11-07T22:27:10.906569+00:00 app[web.1]: [2022-11-07 22:27:10 +0000] [4] [INFO] Shutting down: Master
2022-11-07T22:27:10.906610+00:00 app[web.1]: [2022-11-07 22:27:10 +0000] [4] [INFO] Reason: Worker failed to boot.
2022-11-07T22:27:11.054365+00:00 heroku[web.1]: Process exited with status 3
2022-11-07T22:27:11.153534+00:00 heroku[web.1]: State changed from up to crashed
2022-11-07T22:27:11.963376+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=shows-4-u.herokuapp.com request_id=e0cd543f-98cc-486f-a27e-8f07b97c720f fwd="148.101.185.226" dyno=web.1 connect=0ms service= status=503 bytes= protocol=https
2022-11-07T22:57:34.354131+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=shows-4-u.herokuapp.com request_id=937c2b54-5df2-4e42-b056-1582c6137e0c fwd="148.101.185.226" dyno= connect= service= status=503 bytes= protocol=https
This is my requirements.txt
appdirs==1.4.4
beautifulsoup4==4.11.1
bs4==0.0.1
certifi==2022.9.24
charset-normalizer==2.1.1
click==8.1.3
cssselect==1.2.0
fake-useragent==0.1.14
Flask==2.2.2
gunicorn==20.1.0
idna==3.4
importlib-metadata==5.0.0
itsdangerous==2.1.2
Jinja2==3.1.2
lxml==4.9.1
MarkupSafe==2.1.1
parse==1.19.0
pyee==8.2.2
pyppeteer==1.0.2
pyquery==1.4.3
requests==2.28.1
requests-futures==1.0.0
requests-html==0.10.0
soupsieve==2.3.2.post1
tqdm==4.64.1
urllib3==1.26.12
w3lib==2.0.1
websockets==10.4
Werkzeug==2.2.2
zipp==3.10.0
This is the build log
-----> Using buildpack: heroku/python
-----> Python app detected
-----> Using Python version specified in runtime.txt
!
! A Python security update is available! Upgrade as soon as possible to: python-3.10.8
! See: https://devcenter.heroku.com/articles/python-runtimes
!
-----> Requirements file has been changed, clearing cached dependencies
-----> Installing python-3.10.7
-----> Installing pip 22.3.1, setuptools 63.4.3 and wheel 0.37.1
-----> Installing dependencies with Pipenv 2020.11.15
Installing dependencies from Pipfile.lock (3a221c)...
Collecting beautifulsoup4==4.9.3
Downloading beautifulsoup4-4.9.3-py3-none-any.whl (115 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 115.8/115.8 kB 443.5 kB/s eta 0:00:00
Installing collected packages: beautifulsoup4
WARNING: Ignoring invalid distribution -ertifi (/tmp/build_1fedda59/.heroku/python/lib/python3.10/site-packages)
Successfully installed beautifulsoup4-4.9.3
Ignoring importlib-metadata: markers 'python_version < "3.10"' don't match your environment
-----> Installing SQLite3
-----> Discovering process types
Procfile declares types -> web
-----> Compressing...
Done: 48.2M
-----> Launching...
Released v21
https://shows-4-u.herokuapp.com/ deployed to Heroku
Starting November 28th, 2022, free Heroku Dynos, free Heroku Postgres, and free Heroku Data for Redis® will no longer be available.
If you have apps using any of these resources, you must upgrade to paid plans by this date to ensure your apps continue to run and to retain your data. For students, we will announce a new program by the end of September. Learn more at https://blog.heroku.com/next-chapter```
This question already has answers here:
error cannot import name 'cached_property' from 'werkzeug' when importing [ `from werkzeug.utils import cached_property` ]
(1 answer)
Python3 'Cannot import name 'cached_property'
(6 answers)
Closed 4 months ago.
Hi been looking for a solution for this for hours, need a fresh set of eyes. When deploying my app to heroku the logs have it crashing showing that a worker failed to boot and an import error shortly before that.
Now the current import error is from werkzueg but before I updated flask I was getting an import error from Jninja for 'escape'.
I've also seen that an improper procfile may cause this but I'm pretty sure my procfile is correct.
I'm not setting any ports in either my front/backend and have set the endpoint on the frontend to the url heroku gives me for the build with my specified endpoint.
tldr I don't know what is causing this error
Link to the github for more detailed code https://github.com/Jmumford45/Heart_Prediction_ML
directory outline
app.py is inside Backend
Procfile: web: gunicorn --chdir Backend app:app
error msg:
2022-10-28T23:58:23.563836+00:00 app[web.1]: from werkzeug import cached_property
2022-10-28T23:58:23.563836+00:00 app[web.1]: ImportError: cannot import name 'cached_property' from 'werkzeug' (/app/.heroku/python/lib/python3.10/site-packages/werkzeug/__init__.py)
2022-10-28T23:58:23.563946+00:00 app[web.1]: [2022-10-28 23:58:23 +0000] [9] [INFO] Worker exiting (pid: 9)
2022-10-28T23:58:24.168158+00:00 heroku[web.1]: Process exited with status 3
2022-10-28T23:58:24.234294+00:00 heroku[web.1]: State changed from up to crashed
2022-10-28T23:58:23.897642+00:00 app[web.1]: [2022-10-28 23:58:23 +0000] [4] [WARNING] Worker with pid 9 was terminated due to signal 15
2022-10-28T23:58:23.995629+00:00 app[web.1]: [2022-10-28 23:58:23 +0000] [4] [INFO] Shutting down: Master
2022-10-28T23:58:23.995660+00:00 app[web.1]: [2022-10-28 23:58:23 +0000] [4] [INFO] Reason: Worker failed to boot.
2022-10-28T23:58:47.000000+00:00 app[api]: Build succeeded
2022-10-28T23:58:54.059490+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/favicon.ico" host=heartprediction-app.herokuapp.com request_id=ef722647-4325-4aae-8385-b0ba969c3aa7 fwd="72.111.98.148" dyno= connect= service= status=503 bytes= protocol=https
2022-10-28T23:58:53.791791+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=heartprediction-app.herokuapp.com request_id=fde1ae30-50f0-46d3-b27c-6cd2494ad4d5 fwd="72.111.98.148" dyno= connect= service= status=503 bytes= protocol=https
My heroku app cannot be accessed after a build. The logs show that the web server node and worker node both are listening.
It's a flask app run by gunicorn and it has 2 addons - newrelic and redistogo.
Error:
This site can’t be reached.
appname.herokuapp.com’s server IP address could not be found.
DNS_PROBE_FINISHED_NXDOMAIN
Logs:
```2020-02-05T16:24:31.556201+00:00 heroku[worker.1]: Starting process with command `python worker.py`
2020-02-05T16:24:32.278863+00:00 heroku[worker.1]: State changed from starting to up
2020-02-05T16:24:33.363132+00:00 app[worker.1]: 16:24:33 RQ worker started, version 0.5.1
2020-02-05T16:24:33.364484+00:00 app[worker.1]: 16:24:33
2020-02-05T16:24:33.364574+00:00 app[worker.1]: 16:24:33 *** Listening on high, default, low...
2020-02-05T16:24:35.295791+00:00 heroku[web.1]: Starting process with command `newrelic-admin run-program gunicorn app:server`
2020-02-05T16:24:41.159117+00:00 heroku[web.1]: State changed from starting to up
2020-02-05T16:24:40.959907+00:00 app[web.1]: [2020-02-05 16:24:40 +0000] [4] [INFO] Starting gunicorn 20.0.4
2020-02-05T16:24:40.961836+00:00 app[web.1]: [2020-02-05 16:24:40 +0000] [4] [INFO] Listening at: http://0.0.0.0:21126 (4)
2020-02-05T16:24:40.962097+00:00 app[web.1]: [2020-02-05 16:24:40 +0000] [4] [INFO] Using worker: sync
2020-02-05T16:24:40.971809+00:00 app[web.1]: [2020-02-05 16:24:40 +0000] [12] [INFO] Booting worker with pid: 12
2020-02-05T16:24:41.143051+00:00 app[web.1]: [2020-02-05 16:24:41 +0000] [20] [INFO] Booting worker with pid: 20```
EDIT:
Procfile:
web: newrelic-admin run-program gunicorn app:server
worker: python worker.py
Worker:
import os
import redis
from rq import Worker, Queue, Connection
listen = ['high', 'default', 'low']
redis_url = os.getenv('REDISTOGO_URL', 'redis://localhost:6379')
conn = redis.from_url(redis_url)
if __name__ == '__main__':
with Connection(conn):
worker = Worker(list(map(Queue, listen)))
worker.work()
The app is accessible now. It seems to be downtime with Heroku. Although they haven't reported this in their status page.
Version Info:
"org.apache.storm" % "storm-core" % "1.2.1"
"org.apache.storm" % "storm-kafka-client" % "1.2.1"
I have a storm Topology with 3 bolts(A,B,C), Where the middle bolt takes around 450ms mean time and other two bolts takes less than 1ms.
I am able to run topology with following parallelism hint values:
A: 4
B: 700
C: 10
But when I increase parallelism hint of B to 1200, the topology does not start.
In the topology logs, I see logs to load the executor: B multiple times, like this:
2018-05-18 18:56:37.462 o.a.s.d.executor main [INFO] Loading executor B:[111 111]
2018-05-18 18:56:37.463 o.a.s.d.executor main [INFO] Loaded executor tasks B:[111 111]
2018-05-18 18:56:37.465 o.a.s.d.executor main [INFO] Finished loading executor B:[111 111]
2018-05-18 18:56:37.528 o.a.s.d.executor main [INFO] Loading executor B:[355 355]
2018-05-18 18:56:37.529 o.a.s.d.executor main [INFO] Loaded executor tasks B:[355 355]
2018-05-18 18:56:37.530 o.a.s.d.executor main [INFO] Finished loading executor B:[355 355]
2018-05-18 18:56:37.666 o.a.s.d.executor main [INFO] Loading executor B:[993 993]
2018-05-18 18:56:37.667 o.a.s.d.executor main [INFO] Loaded executor tasks B:[993 993]
2018-05-18 18:56:37.669 o.a.s.d.executor main [INFO] Finished loading executor B:[993 993]
2018-05-18 18:56:37.713 o.a.s.d.executor main [INFO] Loading executor B:[765 765]
2018-05-18 18:56:37.714 o.a.s.d.executor main [INFO] Loaded executor tasks B:[765 765]
But in between worker process get restarted. I don't see any error in topology logs or storm logs. Following are storm logs, when worker gets restart:
2018-05-18 18:51:46.755 o.a.s.d.s.Container SLOT_6700 [INFO] Killing eaf4d8ce-e758-4912-a15d-6dab8cda96d0:766258fe-a604-4385-8eeb-e85cad38b674
2018-05-18 18:51:47.204 o.a.s.d.s.BasicContainer Thread-7 [INFO] Worker Process 766258fe-a604-4385-8eeb-e85cad38b674 exited with code: 143
2018-05-18 18:51:47.766 o.a.s.d.s.Slot SLOT_6700 [INFO] STATE RUNNING msInState: 109081 topo:myTopology-1-1526649581 worker:766258fe-a604-4385-8eeb-e85cad38b674 -> KILL msInState: 0 topo:myTopology-1-1526649581 worker:766258fe-a604-4385-8eeb-e85cad38b674
2018-05-18 18:51:47.766 o.a.s.d.s.Container SLOT_6700 [INFO] GET worker-user for 766258fe-a604-4385-8eeb-e85cad38b674
2018-05-18 18:51:47.774 o.a.s.d.s.Slot SLOT_6700 [WARN] SLOT 6700 all processes are dead...
2018-05-18 18:51:47.775 o.a.s.d.s.Container SLOT_6700 [INFO] Cleaning up eaf4d8ce-e758-4912-a15d-6dab8cda96d0:766258fe-a604-4385-8eeb-e85cad38b674
2018-05-18 18:51:47.775 o.a.s.d.s.Container SLOT_6700 [INFO] GET worker-user for 766258fe-a604-4385-8eeb-e85cad38b674
2018-05-18 18:51:47.775 o.a.s.d.s.AdvancedFSOps SLOT_6700 [INFO] Deleting path /home/saurabh/storm-run/workers/766258fe-a604-4385-8eeb-e85cad38b674/pids/27798
2018-05-18 18:51:47.775 o.a.s.d.s.AdvancedFSOps SLOT_6700 [INFO] Deleting path /home/saurabh/storm-run/workers/766258fe-a604-4385-8eeb-e85cad38b674/heartbeats
2018-05-18 18:51:47.780 o.a.s.d.s.AdvancedFSOps SLOT_6700 [INFO] Deleting path /home/saurabh/storm-run/workers/766258fe-a604-4385-8eeb-e85cad38b674/pids
2018-05-18 18:51:47.780 o.a.s.d.s.AdvancedFSOps SLOT_6700 [INFO] Deleting path /home/saurabh/storm-run/workers/766258fe-a604-4385-8eeb-e85cad38b674/tmp
2018-05-18 18:51:47.781 o.a.s.d.s.AdvancedFSOps SLOT_6700 [INFO] Deleting path /home/saurabh/storm-run/workers/766258fe-a604-4385-8eeb-e85cad38b674
2018-05-18 18:51:47.782 o.a.s.d.s.Container SLOT_6700 [INFO] REMOVE worker-user 766258fe-a604-4385-8eeb-e85cad38b674
2018-05-18 18:51:47.782 o.a.s.d.s.AdvancedFSOps SLOT_6700 [INFO] Deleting path /home/saurabh/storm-run/workers-users/766258fe-a604-4385-8eeb-e85cad38b674
2018-05-18 18:51:47.783 o.a.s.d.s.BasicContainer SLOT_6700 [INFO] Removed Worker ID 766258fe-a604-4385-8eeb-e85cad38b674
2018-05-18 18:51:47.783 o.a.s.l.AsyncLocalizer SLOT_6700 [INFO] Released blob reference myTopology-1-1526649581 6700 Cleaning up BLOB references...
2018-05-18 18:51:47.784 o.a.s.l.AsyncLocalizer SLOT_6700 [INFO] Released blob reference myTopology-1-1526649581 6700 Cleaning up basic files...
2018-05-18 18:51:47.785 o.a.s.d.s.AdvancedFSOps SLOT_6700 [INFO] Deleting path /home/saurabh/storm-run/supervisor/stormdist/myTopology-1-1526649581
2018-05-18 18:51:47.808 o.a.s.d.s.Slot SLOT_6700 [INFO] STATE KILL msInState: 42 topo:myTopology-1-1526649581 worker:null -> EMPTY msInState: 0
This keeps happening and topology never restarts, which used to start perfectly when parallelism hint for bolt: B was 700, there is no other change.
I see one interesting log here is, not yet sure what this means:
Worker Process 766258fe-a604-4385-8eeb-e85cad38b674 exited with code: 143
Any Suggestions?
Edit:
Config:
topology.worker.childopts: -Xms1g -Xmx16g
topology.worker.logwriter.childopts: -Xmx1024m
topology.worker.max.heap.size.mb: 3072.0
worker.childopts: -Xms1g -Xmx16g -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1%ID% -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -XX:+UseG1GC -XX:+AggressiveOpts -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/home/saurabh.mimani/apache-storm-1.2.1/logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M -Dorg.newsclub.net.unix.library.path=/usr/share/specter/uds-lib/
worker.gc.childopts:
worker.heap.memory.mb: 8192
supervisor.childopts: -Xms1g -Xmx16g
Edit:
Logs for strace -fp PID -e trace=read,write,network,signal,ipc in gist.
not yet able to understand it fully, some relevant looking from it:
[pid 3362] open("/usr/lib/locale/UTF-8/LC_CTYPE", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 3362] kill(1487, SIGTERM) = 0
[pid 3362] close(1)
Quick google suggests 143 is the exit code for when the JVM receives a SIGTERM (e.g. Always app Java end with "Exit 143" Ubuntu). You might be running out of memory, or the OS may be killing the process for some other reason. Remember that setting the parallelism hint to 1200 means that you will get 1200 tasks (copies) for bolt B, where you only had 700 before.
I was able to get this running by tweaking following configurations, seems like it was timing out due to nimbus.task.launch.sec, which was set to 120 and it was restarting the worker if it was not started within 120 secs.
Updated value of some of these configs:
drpc.request.timeout.secs: 1600
supervisor.worker.start.timeout.secs: 1200
nimbus.supervisor.timeout.secs: 1200
nimbus.task.launch.secs: 1200
About nimbus.task.launch.sec:
A special timeout used when a task is initially launched. During launch, this is the timeout used until the first heartbeat, overriding nimbus.task.timeout.secs.
A separate timeout exists for launch because there can be quite a bit of overhead to launching new JVM's and configuring them.
I have an issue making my flask app working on heroku.
I'm using a simple flask app along flask-sqlalchemy
I didin't find anything specific on my problem altough I think I understand but can't figure it out how to make it work.
Here is the requirements.txt content:
alembic==0.9.2
click==6.7
Flask==0.12.2
Flask-Migrate==2.0.4
Flask-Script==2.0.5
Flask-SQLAlchemy==2.2
Flask-WTF==0.14.2
gunicorn==19.7.1
itsdangerous==0.24
Jinja2==2.9.6
Mako==1.0.6
MarkupSafe==1.0
python-dateutil==2.6.0
python-editor==1.0.3
six==1.10.0
SQLAlchemy==1.1.10
Werkzeug==0.12.2
WTForms==2.1
Here is the Procfile content:
web: gunicorn main:app
Here is the output of log files in heroku dashboard
2017-06-03T06:59:24.005560+00:00 app[web.1]: return __import__('MySQLdb')
2017-06-03T06:59:24.005561+00:00 app[web.1]: ImportError: No module named MySQLdb
2017-06-03T06:59:24.769740+00:00 heroku[router]: at=info method=GET path="/favicon.ico" host=aqueous-spire-85572.herokuapp.com request_id=965e64b9-20f0-4aa0-8a71-d564068a5c70 fwd="31.5.203.202" dyno=web.1 connect=1ms service=16ms status=404 bytes=386 protocol=https
2017-06-03T07:32:46.321187+00:00 heroku[web.1]: Idling
2017-06-03T07:32:46.321726+00:00 heroku[web.1]: State changed from up to down
2017-06-03T07:32:47.062599+00:00 app[web.1]: [2017-06-03 07:32:47 +0000] [4] [INFO] Handling signal: term
2017-06-03T07:32:47.062699+00:00 app[web.1]: [2017-06-03 07:32:47 +0000] [10] [INFO] Worker exiting (pid: 10)
2017-06-03T07:32:47.063539+00:00 app[web.1]: [2017-06-03 07:32:47 +0000] [9] [INFO] Worker exiting (pid: 9)
2017-06-03T07:32:47.110825+00:00 app[web.1]: [2017-06-03 07:32:47 +0000] [4] [INFO] Shutting down: Master
2017-06-03T07:32:47.055557+00:00 heroku[web.1]: Stopping all processes with SIGTERM
2017-06-03T07:32:47.247466+00:00 heroku[web.1]: Process exited with status 0
The error message tells you, that you haven't installed the MySQLdb module. Add it to your requirements.txt