I'm basically trying to deploy the example in the CherryPy tutorial to Heroku (https://docs.cherrypy.org/en/latest/tutorials.html#tutorial-8-make-it-smoother-with-ajax). When I run the code locally, everything works fine. After deployment to Heroku, I get several error messages:
2020-04-16T17:42:19.422074+00:00 app[web.1]: Traceback (most recent call last):
2020-04-16T17:42:19.422076+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 670, in respond
2020-04-16T17:42:19.422076+00:00 app[web.1]: response.body = self.handler()
2020-04-16T17:42:19.422076+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 217, in __call__
2020-04-16T17:42:19.422077+00:00 app[web.1]: self.body = self.oldhandler(*args, **kwargs)
2020-04-16T17:42:19.422077+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 61, in __call__
2020-04-16T17:42:19.422078+00:00 app[web.1]: return self.callable(*self.args, **self.kwargs)
2020-04-16T17:42:19.422078+00:00 app[web.1]: File "app.py", line 19, in GET
2020-04-16T17:42:19.422078+00:00 app[web.1]: return cherrypy.session['mystring']
2020-04-16T17:42:19.422079+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/cherrypy/__init__.py", line 246, in __getitem__
2020-04-16T17:42:19.422079+00:00 app[web.1]: return child[key]
2020-04-16T17:42:19.422079+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/cherrypy/lib/sessions.py", line 309, in __getitem__
2020-04-16T17:42:19.422080+00:00 app[web.1]: return self._data[key]
2020-04-16T17:42:19.422080+00:00 app[web.1]: KeyError: 'mystring'
2020-04-17T09:24:31.597650+00:00 app[web.1]: File "app.py", line 45, in <module>
2020-04-17T09:24:31.597765+00:00 app[web.1]: webapp.generator = StringGeneratorWebService()
2020-04-17T09:24:31.597768+00:00 app[web.1]: TypeError: expose_() missing 1 required positional argument: 'func'
2020-04-17T09:24:32.000000+00:00 app[api]: Build succeeded
My Procfile simply contains
web: python app.py
My guess is that the problem has something to do with the little css-trick that avoids refreshing the page in order display the result of a request. But, as you can see, I'm totally new to deployment and this is also my first question on stackoverflow--so my apologies in advance in case I'm missing something obvious here. Any help is much appreciated!
Related
I built an MLflow model and call a prediction on a streamlit dashboard, in it work fine in local.
In Heroku, the app which works fine locally failed to send the request online, what am I missing to such deployment?
Below the error code raised.
Procfile :
web: mlflow sagemaker deploy -m mlflow_model/
web: sh setup.sh && streamlit run app.py
Heroku logs:
2022-01-07T10:14:15.759252+00:00 app[web.1]: Traceback (most recent call last):
2022-01-07T10:14:15.759252+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/site-packages/streamlit/script_runner.py", line 354, in _run_script
2022-01-07T10:14:15.759252+00:00 app[web.1]: exec(code, module.__dict__)
2022-01-07T10:14:15.759252+00:00 app[web.1]: File "/app/app.py", line 204, in <module>
2022-01-07T10:14:15.759252+00:00 app[web.1]: main()
2022-01-07T10:14:15.759252+00:00 app[web.1]: File "/app/app.py", line 119, in main
2022-01-07T10:14:15.759253+00:00 app[web.1]: pred = request_prediction(MLFLOW_URI, ml_data)[0]
2022-01-07T10:14:15.759253+00:00 app[web.1]: File "/app/app.py", line 63, in request_prediction
2022-01-07T10:14:15.759253+00:00 app[web.1]: response = requests.request(
2022-01-07T10:14:15.759253+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/site-packages/requests/api.py", line 61, in request
2022-01-07T10:14:15.759253+00:00 app[web.1]: return session.request(method=method, url=url, **kwargs)
2022-01-07T10:14:15.759254+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/site-packages/requests/sessions.py", line 529, in request
2022-01-07T10:14:15.759254+00:00 app[web.1]: resp = self.send(prep, **send_kwargs)
2022-01-07T10:14:15.759254+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/site-packages/requests/sessions.py", line 645, in send
2022-01-07T10:14:15.759254+00:00 app[web.1]: r = adapter.send(request, **kwargs)
2022-01-07T10:14:15.759254+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/site-packages/requests/adapters.py", line 519, in send
2022-01-07T10:14:15.759254+00:00 app[web.1]: raise ConnectionError(e, request=request)
2022-01-07T10:14:15.759259+00:00 app[web.1]: requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /invocations (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f2452d23790>: Failed to establish a new connection: [Errno 111] Connection refused'))
We have in place some guidelines to ensure we can help each other more easily:
do not upload screenshots of code or error logs, but copy-paste and format them
add some code and/or configuration which can help understand the context (i.e. Procfile, application startup code). It is great if you can include a Github repository (if your code is public of course) and ALWAYS avoid including passwords and tokens.
Back to your problem: can you post the Heroku logs and the code to run the streamlit app?
I have an SQLAlchemy database running on a flask app that works fine locally, but as soon as I run it on Heroku I get a strange issue (See stack trace below).
I don't have a model called 'user' so I don't know what's going on. I've made sure that I've run db.create_all().
Creating a row in my table works fine, but it's only once I try selecting some rows do I get an issue (__init__.py):
#app.route('/blog')
def blog():
title = 'Blog'
name = get_name()
blog_text = []
for post in sorted(Post.query.filter_by(category=Post.Category.blog),
key=lambda p: p.posted_at, reverse=True):
content = post.content
blog_text.append(content)
content = '\n'.join(blog_text)
return render_template('blog.html', **locals())
And my 'Post' logic in my models.py:
class Post(db.Model):
__tablename__ = 'user'
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(200), nullable=False)
posted_at = db.Column(db.DateTime, nullable=False, default=datetime.now())
content = db.Column(db.String(200), nullable=False)
content_type = db.Column(db.String(200), default="md", nullable=False) # md / html
category = db.Column(db.String(200), nullable=False, default="blog") # poetry / blog
class Type:
md = "md"
html = "html"
class Category:
poetry = "poetry"
blog = "blog"
...
Here's the error:
2019-07-12T21:59:09.120100+00:00 app[web.1]: 10.63.22.102 - - [12/Jul/2019:21:59:09 +0000] "GET /blog HTTP/1.1" 500 290 "https://secure-savannah-20745.herokuapp.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
2019-07-12T21:59:09.120443+00:00 heroku[router]: at=info method=GET path="/blog" host=secure-savannah-20745.herokuapp.com request_id=c32b2484-c8ff-4910-a4e2-5089eca7c6d3 fwd="98.0.144.26" dyno=web.1 connect=2ms service=66ms status=500 bytes=455 protocol=https
2019-07-12T21:59:13.632588+00:00 app[web.1]: 10.63.22.102 - - [12/Jul/2019:21:59:13 +0000] "GET /projects HTTP/1.1" 200 2342 "https://secure-savannah-20745.herokuapp.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
2019-07-12T21:59:13.837771+00:00 app[web.1]: 10.63.22.102 - - [12/Jul/2019:21:59:13 +0000] "GET /static/circles.js HTTP/1.1" 200 0 "https://secure-savannah-20745.herokuapp.com/projects" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
2019-07-12T21:59:13.838434+00:00 heroku[router]: at=info method=GET path="/static/circles.js" host=secure-savannah-20745.herokuapp.com request_id=8476c020-fc32-47b3-a4e1-7b6faeb115ca fwd="98.0.144.26" dyno=web.1 connect=1ms service=6ms status=200 bytes=5449 protocol=https
2019-07-12T21:59:13.633394+00:00 heroku[router]: at=info method=GET path="/projects" host=secure-savannah-20745.herokuapp.com request_id=faa3d473-7bda-4e39-810b-e3994eac7390 fwd="98.0.144.26" dyno=web.1 connect=1ms service=1494ms status=200 bytes=2504 protocol=https
2019-07-12T21:59:18.074035+00:00 heroku[router]: at=info method=GET path="/blog" host=secure-savannah-20745.herokuapp.com request_id=e9748e89-dcce-43db-8db8-a16c7fe657ab fwd="98.0.144.26" dyno=web.1 connect=3ms service=12ms status=500 bytes=455 protocol=https
2019-07-12T21:59:18.070372+00:00 app[web.1]: [2019-07-12 21:59:18,069] ERROR in app: Exception on /blog [GET]
2019-07-12T21:59:18.070385+00:00 app[web.1]: Traceback (most recent call last):
2019-07-12T21:59:18.070396+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
2019-07-12T21:59:18.070398+00:00 app[web.1]: cursor, statement, parameters, context
2019-07-12T21:59:18.070403+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 550, in do_execute
2019-07-12T21:59:18.070405+00:00 app[web.1]: cursor.execute(statement, parameters)
2019-07-12T21:59:18.070407+00:00 app[web.1]: sqlite3.OperationalError: no such table: user
2019-07-12T21:59:18.070410+00:00 app[web.1]:
2019-07-12T21:59:18.070412+00:00 app[web.1]: The above exception was the direct cause of the following exception:
2019-07-12T21:59:18.070414+00:00 app[web.1]:
2019-07-12T21:59:18.070416+00:00 app[web.1]: Traceback (most recent call last):
2019-07-12T21:59:18.070418+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app
2019-07-12T21:59:18.070421+00:00 app[web.1]: response = self.full_dispatch_request()
2019-07-12T21:59:18.070423+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 1815, in full_dispatch_request
2019-07-12T21:59:18.070425+00:00 app[web.1]: rv = self.handle_user_exception(e)
2019-07-12T21:59:18.070427+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 1718, in handle_user_exception
2019-07-12T21:59:18.070430+00:00 app[web.1]: reraise(exc_type, exc_value, tb)
2019-07-12T21:59:18.070432+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
2019-07-12T21:59:18.070434+00:00 app[web.1]: raise value
2019-07-12T21:59:18.070436+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
2019-07-12T21:59:18.070438+00:00 app[web.1]: rv = self.dispatch_request()
2019-07-12T21:59:18.070446+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
2019-07-12T21:59:18.070450+00:00 app[web.1]: return self.view_functions[rule.endpoint](**req.view_args)
2019-07-12T21:59:18.070452+00:00 app[web.1]: File "/app/main.py", line 65, in blog
2019-07-12T21:59:18.070455+00:00 app[web.1]: key=lambda p: p.posted_at, reverse=True):
2019-07-12T21:59:18.070457+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3324, in __iter__
2019-07-12T21:59:18.070459+00:00 app[web.1]: return self._execute_and_instances(context)
2019-07-12T21:59:18.070461+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3349, in _execute_and_instances
2019-07-12T21:59:18.070463+00:00 app[web.1]: result = conn.execute(querycontext.statement, self._params)
2019-07-12T21:59:18.070466+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute
2019-07-12T21:59:18.070468+00:00 app[web.1]: return meth(self, multiparams, params)
2019-07-12T21:59:18.070470+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
2019-07-12T21:59:18.070472+00:00 app[web.1]: return connection._execute_clauseelement(self, multiparams, params)
2019-07-12T21:59:18.070474+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
2019-07-12T21:59:18.070477+00:00 app[web.1]: distilled_params,
2019-07-12T21:59:18.070479+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
2019-07-12T21:59:18.070481+00:00 app[web.1]: e, statement, parameters, cursor, context
2019-07-12T21:59:18.070484+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
2019-07-12T21:59:18.070486+00:00 app[web.1]: util.raise_from_cause(sqlalchemy_exception, exc_info)
2019-07-12T21:59:18.070488+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 399, in raise_from_cause
2019-07-12T21:59:18.070491+00:00 app[web.1]: reraise(type(exception), exception, tb=exc_tb, cause=cause)
2019-07-12T21:59:18.070493+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
2019-07-12T21:59:18.070495+00:00 app[web.1]: raise value.with_traceback(tb)
2019-07-12T21:59:18.070498+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
2019-07-12T21:59:18.070500+00:00 app[web.1]: cursor, statement, parameters, context
2019-07-12T21:59:18.070502+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 550, in do_execute
2019-07-12T21:59:18.070504+00:00 app[web.1]: cursor.execute(statement, parameters)
2019-07-12T21:59:18.070507+00:00 app[web.1]: sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: user
2019-07-12T21:59:18.070513+00:00 app[web.1]: [SQL: SELECT user.id AS user_id, user.name AS user_name, user.posted_at AS user_posted_at, user.content AS user_content, user.content_type AS user_content_type, user.category AS user_category
2019-07-12T21:59:18.070515+00:00 app[web.1]: FROM user
2019-07-12T21:59:18.070517+00:00 app[web.1]: WHERE user.category = ?]
2019-07-12T21:59:18.070520+00:00 app[web.1]: [parameters: ('blog',)]
2019-07-12T21:59:18.070661+00:00 app[web.1]: (Background on this error at: http://sqlalche.me/e/e3q8)
2019-07-12T21:59:18.072919+00:00 app[web.1]: 10.63.22.102 - - [12/Jul/2019:21:59:18 +0000] "GET /blog HTTP/1.1" 500 290 "https://secure-savannah-20745.herokuapp.com/projects" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
You missed that your table is named user in your model:
class Post(db.Model):
__tablename__ = 'user'
# ^^^^^^
The reason it doesn't exist is because Heroku doesn't keep your filesystem changes. You are using a SQLite database:
... app[web.1]: sqlite3.OperationalError: no such table: user
# ^^^^^^^
but a sqlite database is stored on the filesystem of your Heroku dyno. Heroku cleans up (cycles) dynos all the time and spins up a new dyno for you for new requests. From their documentation on how Heroku works:
Changes to the filesystem on one dyno are not propagated to other dynos and are not persisted across deploys and dyno restarts. A better and more scalable approach is to use a shared resource such as a database or queue.
Each time a new dyno is spun up to handle your requests, you'll have a new, clean filesystem without your sqlite database, and so an empty database is created again without any tables in it.
Bottom line: You can't use sqlite on Heroku.
Use a Postgres database instead, see the Heroku Postgres documentation. Don't worry, there is a free tier you can use. When you provision the database as an add-on, the DATABASE_URL config var is set for you, which you can read from your app environment.
I usually configure Flask apps that are designed to run on Heroku to read the environment variable but to fall back to a SQLite database when the environment variable is not set:
import os
from pathlib import Path
_project_root = Path(__file__).resolve().parent.parent
_default_sqlite_db = _project_root / "database.db"
SQLALCHEMY_DATABASE_URI = os.environ.get(
"DATABASE_URL", f"sqlite:///{_default_sqlite_db}"
)
This makes it easy to develop the app locally on SQLite, or set up a local Postgres database for integration testing, and then deploy to Heroku for staging or production.
My server on Heroku [seemingly] randomly returns H18 errors.
I use:
Gunicorn (I tried both sync and gevent workers)
Flask
Flask-Restful
If I repeat the same request several times, I occasionally get an H18. I noticed that the most often I get such a response when an exception is raised in my code. I have a handler defined that returns a valid HTTP error response and most of the time it succeeds. Sometimes it fails though - for the reason that I cannot understand.
Here's how the "Bad situation" log looks like:
2018-03-10T17:50:51.875115+00:00 app[web.1]: [2018-03-10 17:50:51 +0000] [11] [DEBUG] PUT /api/v1/views/270
2018-03-10T17:50:51.879035+00:00 app[web.1]: PPMException: Token expired
2018-03-10T17:50:51.879442+00:00 app[web.1]: Traceback (most recent call last):
2018-03-10T17:50:51.879445+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
2018-03-10T17:50:51.879447+00:00 app[web.1]: rv = self.dispatch_request()
2018-03-10T17:50:51.879449+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
2018-03-10T17:50:51.879451+00:00 app[web.1]: return self.view_functions[rule.endpoint](**req.view_args)
2018-03-10T17:50:51.879452+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask_restful/__init__.py", line 480, in wrapper
2018-03-10T17:50:51.879456+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/views.py", line 84, in view
2018-03-10T17:50:51.879454+00:00 app[web.1]: resp = resource(*args, **kwargs)
2018-03-10T17:50:51.879458+00:00 app[web.1]: return self.dispatch_request(*args, **kwargs)
2018-03-10T17:50:51.879464+00:00 app[web.1]: resp = meth(*args, **kwargs)
2018-03-10T17:50:51.879466+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask_httpauth.py", line 88, in decorated
2018-03-10T17:50:51.879463+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask_restful/__init__.py", line 595, in dispatch_request
2018-03-10T17:50:51.879469+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask_httpauth.py", line 125, in authenticate
2018-03-10T17:50:51.879468+00:00 app[web.1]: if not self.authenticate(auth, password):
2018-03-10T17:50:51.879471+00:00 app[web.1]: return self.verify_password_callback(username, client_password)
2018-03-10T17:50:51.879473+00:00 app[web.1]: File "/app/app/api_security.py", line 31, in verify_password
2018-03-10T17:50:51.879477+00:00 app[web.1]: PPMException: Token expired
2018-03-10T17:50:51.879475+00:00 app[web.1]: raise PPMException("SECURITY_TOKEN_EXPIRED", http_code=401)
2018-03-10T17:50:51.879479+00:00 app[web.1]:
2018-03-10T17:50:51.881229+00:00 app[web.1]: 10.189.169.232 - eyJleHAiOjE1MjA3MDQyNTAsImlhdCI6MTUyMDcwNDI0MCwiYWxnIjoiSFMyNTYifQ.eyJ1c2VyX2lkIjoiMyIsIm9yaWdpbmFsX2lzc3VlX2RhdGUiOiIyMDE4LTAzLTEwVDE3OjQ5OjI0Ljk3NTkwMCJ9.mB_NCtrGeDSq0ATs-Zl-Fg8qYRCx8YyIkjVdaLi_O6c
[10/Mar/2018:17:50:51 +0000] "PUT /api/v1/views/270 HTTP/1.1" 401 91 "http://localhost:3000/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
2018-03-10T17:50:51.881448+00:00 app[web.1]: [2018-03-10 17:50:51 +0000] [11] [DEBUG] Closing connection.
2018-03-10T17:50:51.975825+00:00 heroku[router]: sock=backend at=error code=H18 desc="Server Request Interrupted" method=PUT path="/api/v1/views/270" host=***-dev.herokuapp.com request_id=68b4ddb1-19b4-4753-9b3f-395c80fc777a fwd="81.164.187.197" dyno=web.1 connect=2ms service=106ms status=503 bytes=483 protocol=https
vs. how the same exception looks without H18:
2018-03-10T17:53:22.660154+00:00 app[web.1]: [2018-03-10 17:53:22 +0000] [10] [DEBUG] PUT /api/v1/views/270
2018-03-10T17:53:22.667461+00:00 app[web.1]: PPMException: Token expired
2018-03-10T17:53:22.669037+00:00 app[web.1]: Traceback (most recent call last):
2018-03-10T17:53:22.669041+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
2018-03-10T17:53:22.669042+00:00 app[web.1]: rv = self.dispatch_request()
2018-03-10T17:53:22.669044+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
2018-03-10T17:53:22.669046+00:00 app[web.1]: return self.view_functions[rule.endpoint](**req.view_args)
2018-03-10T17:53:22.669048+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask_restful/__init__.py", line 480, in wrapper
2018-03-10T17:53:22.669050+00:00 app[web.1]: resp = resource(*args, **kwargs)
2018-03-10T17:53:22.669055+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/views.py", line 84, in view
2018-03-10T17:53:22.669057+00:00 app[web.1]: return self.dispatch_request(*args, **kwargs)
2018-03-10T17:53:22.669059+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask_restful/__init__.py", line 595, in dispatch_request
2018-03-10T17:53:22.669060+00:00 app[web.1]: resp = meth(*args, **kwargs)
2018-03-10T17:53:22.669062+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask_httpauth.py", line 88, in decorated
2018-03-10T17:53:22.669064+00:00 app[web.1]: if not self.authenticate(auth, password):
2018-03-10T17:53:22.669070+00:00 app[web.1]: File "/app/app/api_security.py", line 31, in verify_password
2018-03-10T17:53:22.669075+00:00 app[web.1]:
2018-03-10T17:53:22.669066+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask_httpauth.py", line 125, in authenticate
2018-03-10T17:53:22.669068+00:00 app[web.1]: return self.verify_password_callback(username, client_password)
2018-03-10T17:53:22.669071+00:00 app[web.1]: raise PPMException("SECURITY_TOKEN_EXPIRED", http_code=401)
2018-03-10T17:53:22.669073+00:00 app[web.1]: PPMException: Token expired
2018-03-10T17:53:22.677909+00:00 app[web.1]: 10.31.61.151 - eyJleHAiOjE1MjA3MDQyNTAsImlhdCI6MTUyMDcwNDI0MCwiYWxnIjoiSFMyNTYifQ.eyJ1c2VyX2lkIjoiMyIsIm9yaWdpbmFsX2lzc3VlX2RhdGUiOiIyMDE4LTAzLTEwVDE3OjQ5OjI0Ljk3NTkwMCJ9.mB_NCtrGeDSq0ATs-Zl-Fg8qYRCx8YyIkjVdaLi_O6c [10/Mar/2018:17:53:22 +0000] "PUT /api/v1/views/270 HTTP/1.1" 401 91 "http://localhost:3000/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
***2018-03-10T17:53:22.678478+00:00 heroku[router]: at=info method=PUT path="/api/v1/views/270" host=***-dev.herokuapp.com request_id=9e22f6a9-c980-478f-9ff1-bc8bbd4fe8c7 fwd="81.164.187.197" dyno=web.1 connect=0ms service=19ms status=401 bytes=483 protocol=https
2018-03-10T17:53:22.678989+00:00 app[web.1]: [2018-03-10 17:53:22 +0000] [10] [DEBUG] Closing connection.
The only difference I see is that in the first case I first see Gunicorn saying "Closing connection" and then the heroku router saying H18, while in the second case heroku router reports the response and then Gunicorn is closing the connection. If Gunicorn is closing connection prematurely, is there a way for me to control it from the application?
I think I got this figured out.
The reason seems to be that there were still request data left unread, so I added "request.stream.read()" in my error handling code.
I'm using Spark (pyspark) to read in data and have ran into a snag as some of my data is now in .gz format.
%pyspark
data = sc.textFile("s3://mybucket.file.gz")
data.first()
Traceback (most recent call last):
File "/tmp/zeppelin_pyspark-5205743886772607083.py", line 267, in <module>
raise Exception(traceback.format_exc())
Exception: Traceback (most recent call last):
File "/tmp/zeppelin_pyspark-5205743886772607083.py", line 260, in <module>
exec(code)
File "<stdin>", line 1, in <module>
File "/usr/lib/spark/python/pyspark/rdd.py", line 1041, in count
return self.mapPartitions(lambda i: [sum(1 for _ in i)]).sum()
File "/usr/lib/spark/python/pyspark/rdd.py", line 1032, in sum
return self.mapPartitions(lambda x: [sum(x)]).fold(0, operator.add)
File "/usr/lib/spark/python/pyspark/rdd.py", line 906, in fold
vals = self.mapPartitions(func).collect()
File "/usr/lib/spark/python/pyspark/rdd.py", line 809, in collect
port = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/usr/lib/spark/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
format(target_id, ".", name), value)
Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 31.0 failed 4 times, most recent failure: Lost task 0.3 in stage 31.0 (TID 270, ip-172-16-238-231.us-west-1.compute.internal, executor 17): java.io.IOException: incorrect header check
at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
Any thoughts on how to read this in and uncompress?
Strangely enough, all I had to do is remove the "gz" off the end and it worked.
My Rails app has been running fine until this morning. I tried restarting the app and here is the error i'm getting before i even hit a page. Out of Memory error on the Postgres connection! I can connect fine to the database from my dev machine. I have a Dev account with Heroku Postgres. I also ran that that SQL statement, and it was fine it should just return 1.
The config var DATABASE_URL is set correctly.
Any ideas?
thx,
matt
app[web.1]: => Booting WEBrick
app[web.1]: => Rails 3.2.13 application starting in production on http://0.0.0.0:23827
app[web.1]: => Call with -d to detach
app[web.1]: => Ctrl-C to shutdown server
app[web.1]: Exiting
app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.13/lib/active_record/connection_adapters/postgresql_adapter.rb:1161:in `exec': PG::OutOfMemory: ERROR: out of memory (ActiveRecord::StatementInvalid)
app[web.1]: DETAIL: Failed on request of size 96.
app[web.1]: : SELECT COUNT(*)
app[web.1]: FROM pg_class c
app[web.1]: LEFT JOIN pg_namespace n ON n.oid = c.relnamespace
app[web.1]: WHERE c.relkind in ('v','r')
app[web.1]: AND c.relname = 'spree_shipments'
app[web.1]: AND n.nspname = ANY (current_schemas(false))
app[web.1]: from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2