I built an MLflow model and call a prediction on a streamlit dashboard, in it work fine in local.
In Heroku, the app which works fine locally failed to send the request online, what am I missing to such deployment?
Below the error code raised.
Procfile :
web: mlflow sagemaker deploy -m mlflow_model/
web: sh setup.sh && streamlit run app.py
Heroku logs:
2022-01-07T10:14:15.759252+00:00 app[web.1]: Traceback (most recent call last):
2022-01-07T10:14:15.759252+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/site-packages/streamlit/script_runner.py", line 354, in _run_script
2022-01-07T10:14:15.759252+00:00 app[web.1]: exec(code, module.__dict__)
2022-01-07T10:14:15.759252+00:00 app[web.1]: File "/app/app.py", line 204, in <module>
2022-01-07T10:14:15.759252+00:00 app[web.1]: main()
2022-01-07T10:14:15.759252+00:00 app[web.1]: File "/app/app.py", line 119, in main
2022-01-07T10:14:15.759253+00:00 app[web.1]: pred = request_prediction(MLFLOW_URI, ml_data)[0]
2022-01-07T10:14:15.759253+00:00 app[web.1]: File "/app/app.py", line 63, in request_prediction
2022-01-07T10:14:15.759253+00:00 app[web.1]: response = requests.request(
2022-01-07T10:14:15.759253+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/site-packages/requests/api.py", line 61, in request
2022-01-07T10:14:15.759253+00:00 app[web.1]: return session.request(method=method, url=url, **kwargs)
2022-01-07T10:14:15.759254+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/site-packages/requests/sessions.py", line 529, in request
2022-01-07T10:14:15.759254+00:00 app[web.1]: resp = self.send(prep, **send_kwargs)
2022-01-07T10:14:15.759254+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/site-packages/requests/sessions.py", line 645, in send
2022-01-07T10:14:15.759254+00:00 app[web.1]: r = adapter.send(request, **kwargs)
2022-01-07T10:14:15.759254+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/site-packages/requests/adapters.py", line 519, in send
2022-01-07T10:14:15.759254+00:00 app[web.1]: raise ConnectionError(e, request=request)
2022-01-07T10:14:15.759259+00:00 app[web.1]: requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /invocations (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f2452d23790>: Failed to establish a new connection: [Errno 111] Connection refused'))
We have in place some guidelines to ensure we can help each other more easily:
do not upload screenshots of code or error logs, but copy-paste and format them
add some code and/or configuration which can help understand the context (i.e. Procfile, application startup code). It is great if you can include a Github repository (if your code is public of course) and ALWAYS avoid including passwords and tokens.
Back to your problem: can you post the Heroku logs and the code to run the streamlit app?
Related
I'm basically trying to deploy the example in the CherryPy tutorial to Heroku (https://docs.cherrypy.org/en/latest/tutorials.html#tutorial-8-make-it-smoother-with-ajax). When I run the code locally, everything works fine. After deployment to Heroku, I get several error messages:
2020-04-16T17:42:19.422074+00:00 app[web.1]: Traceback (most recent call last):
2020-04-16T17:42:19.422076+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 670, in respond
2020-04-16T17:42:19.422076+00:00 app[web.1]: response.body = self.handler()
2020-04-16T17:42:19.422076+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 217, in __call__
2020-04-16T17:42:19.422077+00:00 app[web.1]: self.body = self.oldhandler(*args, **kwargs)
2020-04-16T17:42:19.422077+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 61, in __call__
2020-04-16T17:42:19.422078+00:00 app[web.1]: return self.callable(*self.args, **self.kwargs)
2020-04-16T17:42:19.422078+00:00 app[web.1]: File "app.py", line 19, in GET
2020-04-16T17:42:19.422078+00:00 app[web.1]: return cherrypy.session['mystring']
2020-04-16T17:42:19.422079+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/cherrypy/__init__.py", line 246, in __getitem__
2020-04-16T17:42:19.422079+00:00 app[web.1]: return child[key]
2020-04-16T17:42:19.422079+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/cherrypy/lib/sessions.py", line 309, in __getitem__
2020-04-16T17:42:19.422080+00:00 app[web.1]: return self._data[key]
2020-04-16T17:42:19.422080+00:00 app[web.1]: KeyError: 'mystring'
2020-04-17T09:24:31.597650+00:00 app[web.1]: File "app.py", line 45, in <module>
2020-04-17T09:24:31.597765+00:00 app[web.1]: webapp.generator = StringGeneratorWebService()
2020-04-17T09:24:31.597768+00:00 app[web.1]: TypeError: expose_() missing 1 required positional argument: 'func'
2020-04-17T09:24:32.000000+00:00 app[api]: Build succeeded
My Procfile simply contains
web: python app.py
My guess is that the problem has something to do with the little css-trick that avoids refreshing the page in order display the result of a request. But, as you can see, I'm totally new to deployment and this is also my first question on stackoverflow--so my apologies in advance in case I'm missing something obvious here. Any help is much appreciated!
I have an SQLAlchemy database running on a flask app that works fine locally, but as soon as I run it on Heroku I get a strange issue (See stack trace below).
I don't have a model called 'user' so I don't know what's going on. I've made sure that I've run db.create_all().
Creating a row in my table works fine, but it's only once I try selecting some rows do I get an issue (__init__.py):
#app.route('/blog')
def blog():
title = 'Blog'
name = get_name()
blog_text = []
for post in sorted(Post.query.filter_by(category=Post.Category.blog),
key=lambda p: p.posted_at, reverse=True):
content = post.content
blog_text.append(content)
content = '\n'.join(blog_text)
return render_template('blog.html', **locals())
And my 'Post' logic in my models.py:
class Post(db.Model):
__tablename__ = 'user'
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(200), nullable=False)
posted_at = db.Column(db.DateTime, nullable=False, default=datetime.now())
content = db.Column(db.String(200), nullable=False)
content_type = db.Column(db.String(200), default="md", nullable=False) # md / html
category = db.Column(db.String(200), nullable=False, default="blog") # poetry / blog
class Type:
md = "md"
html = "html"
class Category:
poetry = "poetry"
blog = "blog"
...
Here's the error:
2019-07-12T21:59:09.120100+00:00 app[web.1]: 10.63.22.102 - - [12/Jul/2019:21:59:09 +0000] "GET /blog HTTP/1.1" 500 290 "https://secure-savannah-20745.herokuapp.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
2019-07-12T21:59:09.120443+00:00 heroku[router]: at=info method=GET path="/blog" host=secure-savannah-20745.herokuapp.com request_id=c32b2484-c8ff-4910-a4e2-5089eca7c6d3 fwd="98.0.144.26" dyno=web.1 connect=2ms service=66ms status=500 bytes=455 protocol=https
2019-07-12T21:59:13.632588+00:00 app[web.1]: 10.63.22.102 - - [12/Jul/2019:21:59:13 +0000] "GET /projects HTTP/1.1" 200 2342 "https://secure-savannah-20745.herokuapp.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
2019-07-12T21:59:13.837771+00:00 app[web.1]: 10.63.22.102 - - [12/Jul/2019:21:59:13 +0000] "GET /static/circles.js HTTP/1.1" 200 0 "https://secure-savannah-20745.herokuapp.com/projects" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
2019-07-12T21:59:13.838434+00:00 heroku[router]: at=info method=GET path="/static/circles.js" host=secure-savannah-20745.herokuapp.com request_id=8476c020-fc32-47b3-a4e1-7b6faeb115ca fwd="98.0.144.26" dyno=web.1 connect=1ms service=6ms status=200 bytes=5449 protocol=https
2019-07-12T21:59:13.633394+00:00 heroku[router]: at=info method=GET path="/projects" host=secure-savannah-20745.herokuapp.com request_id=faa3d473-7bda-4e39-810b-e3994eac7390 fwd="98.0.144.26" dyno=web.1 connect=1ms service=1494ms status=200 bytes=2504 protocol=https
2019-07-12T21:59:18.074035+00:00 heroku[router]: at=info method=GET path="/blog" host=secure-savannah-20745.herokuapp.com request_id=e9748e89-dcce-43db-8db8-a16c7fe657ab fwd="98.0.144.26" dyno=web.1 connect=3ms service=12ms status=500 bytes=455 protocol=https
2019-07-12T21:59:18.070372+00:00 app[web.1]: [2019-07-12 21:59:18,069] ERROR in app: Exception on /blog [GET]
2019-07-12T21:59:18.070385+00:00 app[web.1]: Traceback (most recent call last):
2019-07-12T21:59:18.070396+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
2019-07-12T21:59:18.070398+00:00 app[web.1]: cursor, statement, parameters, context
2019-07-12T21:59:18.070403+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 550, in do_execute
2019-07-12T21:59:18.070405+00:00 app[web.1]: cursor.execute(statement, parameters)
2019-07-12T21:59:18.070407+00:00 app[web.1]: sqlite3.OperationalError: no such table: user
2019-07-12T21:59:18.070410+00:00 app[web.1]:
2019-07-12T21:59:18.070412+00:00 app[web.1]: The above exception was the direct cause of the following exception:
2019-07-12T21:59:18.070414+00:00 app[web.1]:
2019-07-12T21:59:18.070416+00:00 app[web.1]: Traceback (most recent call last):
2019-07-12T21:59:18.070418+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app
2019-07-12T21:59:18.070421+00:00 app[web.1]: response = self.full_dispatch_request()
2019-07-12T21:59:18.070423+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 1815, in full_dispatch_request
2019-07-12T21:59:18.070425+00:00 app[web.1]: rv = self.handle_user_exception(e)
2019-07-12T21:59:18.070427+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 1718, in handle_user_exception
2019-07-12T21:59:18.070430+00:00 app[web.1]: reraise(exc_type, exc_value, tb)
2019-07-12T21:59:18.070432+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
2019-07-12T21:59:18.070434+00:00 app[web.1]: raise value
2019-07-12T21:59:18.070436+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
2019-07-12T21:59:18.070438+00:00 app[web.1]: rv = self.dispatch_request()
2019-07-12T21:59:18.070446+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
2019-07-12T21:59:18.070450+00:00 app[web.1]: return self.view_functions[rule.endpoint](**req.view_args)
2019-07-12T21:59:18.070452+00:00 app[web.1]: File "/app/main.py", line 65, in blog
2019-07-12T21:59:18.070455+00:00 app[web.1]: key=lambda p: p.posted_at, reverse=True):
2019-07-12T21:59:18.070457+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3324, in __iter__
2019-07-12T21:59:18.070459+00:00 app[web.1]: return self._execute_and_instances(context)
2019-07-12T21:59:18.070461+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3349, in _execute_and_instances
2019-07-12T21:59:18.070463+00:00 app[web.1]: result = conn.execute(querycontext.statement, self._params)
2019-07-12T21:59:18.070466+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute
2019-07-12T21:59:18.070468+00:00 app[web.1]: return meth(self, multiparams, params)
2019-07-12T21:59:18.070470+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
2019-07-12T21:59:18.070472+00:00 app[web.1]: return connection._execute_clauseelement(self, multiparams, params)
2019-07-12T21:59:18.070474+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
2019-07-12T21:59:18.070477+00:00 app[web.1]: distilled_params,
2019-07-12T21:59:18.070479+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
2019-07-12T21:59:18.070481+00:00 app[web.1]: e, statement, parameters, cursor, context
2019-07-12T21:59:18.070484+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
2019-07-12T21:59:18.070486+00:00 app[web.1]: util.raise_from_cause(sqlalchemy_exception, exc_info)
2019-07-12T21:59:18.070488+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 399, in raise_from_cause
2019-07-12T21:59:18.070491+00:00 app[web.1]: reraise(type(exception), exception, tb=exc_tb, cause=cause)
2019-07-12T21:59:18.070493+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
2019-07-12T21:59:18.070495+00:00 app[web.1]: raise value.with_traceback(tb)
2019-07-12T21:59:18.070498+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
2019-07-12T21:59:18.070500+00:00 app[web.1]: cursor, statement, parameters, context
2019-07-12T21:59:18.070502+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 550, in do_execute
2019-07-12T21:59:18.070504+00:00 app[web.1]: cursor.execute(statement, parameters)
2019-07-12T21:59:18.070507+00:00 app[web.1]: sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: user
2019-07-12T21:59:18.070513+00:00 app[web.1]: [SQL: SELECT user.id AS user_id, user.name AS user_name, user.posted_at AS user_posted_at, user.content AS user_content, user.content_type AS user_content_type, user.category AS user_category
2019-07-12T21:59:18.070515+00:00 app[web.1]: FROM user
2019-07-12T21:59:18.070517+00:00 app[web.1]: WHERE user.category = ?]
2019-07-12T21:59:18.070520+00:00 app[web.1]: [parameters: ('blog',)]
2019-07-12T21:59:18.070661+00:00 app[web.1]: (Background on this error at: http://sqlalche.me/e/e3q8)
2019-07-12T21:59:18.072919+00:00 app[web.1]: 10.63.22.102 - - [12/Jul/2019:21:59:18 +0000] "GET /blog HTTP/1.1" 500 290 "https://secure-savannah-20745.herokuapp.com/projects" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
You missed that your table is named user in your model:
class Post(db.Model):
__tablename__ = 'user'
# ^^^^^^
The reason it doesn't exist is because Heroku doesn't keep your filesystem changes. You are using a SQLite database:
... app[web.1]: sqlite3.OperationalError: no such table: user
# ^^^^^^^
but a sqlite database is stored on the filesystem of your Heroku dyno. Heroku cleans up (cycles) dynos all the time and spins up a new dyno for you for new requests. From their documentation on how Heroku works:
Changes to the filesystem on one dyno are not propagated to other dynos and are not persisted across deploys and dyno restarts. A better and more scalable approach is to use a shared resource such as a database or queue.
Each time a new dyno is spun up to handle your requests, you'll have a new, clean filesystem without your sqlite database, and so an empty database is created again without any tables in it.
Bottom line: You can't use sqlite on Heroku.
Use a Postgres database instead, see the Heroku Postgres documentation. Don't worry, there is a free tier you can use. When you provision the database as an add-on, the DATABASE_URL config var is set for you, which you can read from your app environment.
I usually configure Flask apps that are designed to run on Heroku to read the environment variable but to fall back to a SQLite database when the environment variable is not set:
import os
from pathlib import Path
_project_root = Path(__file__).resolve().parent.parent
_default_sqlite_db = _project_root / "database.db"
SQLALCHEMY_DATABASE_URI = os.environ.get(
"DATABASE_URL", f"sqlite:///{_default_sqlite_db}"
)
This makes it easy to develop the app locally on SQLite, or set up a local Postgres database for integration testing, and then deploy to Heroku for staging or production.
I followed the examples:
from kubernetes import client, config
config.load_kube_config()
v1 = client.CoreV1Api()
print("Listing pods with their IPs:")
ret = v1.list_pod_for_all_namespaces(watch=False)
for i in ret.items:
print("%s\t%s\t%s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))
But always got this error:
2018-08-28 23:03:48,818 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),)': /api/v1/pods?watch=False
...ommit some retry logs
Traceback (most recent call last):
File "/Users/maxpeng/.pyenv/versions/3.5.0/lib/python3.5/site-packages/urllib3/contrib/pyopenssl.py", line 444, in wrap_socket
cnx.do_handshake()
File "/Users/maxpeng/.pyenv/versions/3.5.0/lib/python3.5/site-packages/OpenSSL/SSL.py", line 1907, in do_handshake
self._raise_ssl_error(self._ssl, result)
File "/Users/maxpeng/.pyenv/versions/3.5.0/lib/python3.5/site-packages/OpenSSL/SSL.py", line 1639, in _raise_ssl_error
_raise_current_error()
File "/Users/maxpeng/.pyenv/versions/3.5.0/lib/python3.5/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/maxpeng/.pyenv/versions/3.5.0/lib/python3.5/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/Users/maxpeng/.pyenv/versions/3.5.0/lib/python3.5/site-packages/urllib3/connectionpool.py", line 343, in _make_request
self._validate_conn(conn)
File "/Users/maxpeng/.pyenv/versions/3.5.0/lib/python3.5/site-packages/urllib3/connectionpool.py", line 849, in _validate_conn
conn.connect()
File "/Users/maxpeng/.pyenv/versions/3.5.0/lib/python3.5/site-packages/urllib3/connection.py", line 356, in connect
ssl_context=context)
File "/Users/maxpeng/.pyenv/versions/3.5.0/lib/python3.5/site-packages/urllib3/util/ssl_.py", line 372, in ssl_wrap_socket
return context.wrap_socket(sock)
File "/Users/maxpeng/.pyenv/versions/3.5.0/lib/python3.5/site-packages/urllib3/contrib/pyopenssl.py", line 450, in wrap_socket
raise ssl.SSLError('bad handshake: %r' % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/maxpeng/.pyenv/versions/3.5.0/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py", line 13608, in list_pod_for_all_namespaces
(data) = self.list_pod_for_all_namespaces_with_http_info(**kwargs)
File "/Users/maxpeng/.pyenv/versions/3.5.0/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py", line 13705, in list_pod_for_all_namespaces_with_http_info
collection_formats=collection_formats)
File "/Users/maxpeng/.pyenv/versions/3.5.0/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 321, in call_api
_return_http_data_only, collection_formats, _preload_content, _request_timeout)
File "/Users/maxpeng/.pyenv/versions/3.5.0/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 155, in __call_api
_request_timeout=_request_timeout)
File "/Users/maxpeng/.pyenv/versions/3.5.0/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 342, in request
headers=headers)
File "/Users/maxpeng/.pyenv/versions/3.5.0/lib/python3.5/site-packages/kubernetes/client/rest.py", line 231, in GET
query_params=query_params)
File "/Users/maxpeng/.pyenv/versions/3.5.0/lib/python3.5/site-packages/kubernetes/client/rest.py", line 205, in request
headers=headers)
File "/Users/maxpeng/.pyenv/versions/3.5.0/lib/python3.5/site-packages/urllib3/request.py", line 68, in request
**urlopen_kw)
File "/Users/maxpeng/.pyenv/versions/3.5.0/lib/python3.5/site-packages/urllib3/request.py", line 89, in request_encode_url
return self.urlopen(method, url, **extra_kw)
File "/Users/maxpeng/.pyenv/versions/3.5.0/lib/python3.5/site-packages/urllib3/poolmanager.py", line 322, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/Users/maxpeng/.pyenv/versions/3.5.0/lib/python3.5/site-packages/urllib3/connectionpool.py", line 667, in urlopen
**response_kw)
File "/Users/maxpeng/.pyenv/versions/3.5.0/lib/python3.5/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/Users/maxpeng/.pyenv/versions/3.5.0/lib/python3.5/site-packages/urllib3/util/retry.py", line 398, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='xx.xx.xx.xx', port=xxxx): Max retries exceeded with url: /api/v1/pods?watch=False (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),))
Mac OSX 10.13.6
kubectl version
Client Version: version.Info{Major:"1", Minor:"11",
GitVersion:"v1.11.2",
GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239",
GitTreeState:"clean", BuildDate:"2018-08-08T16:31:10Z",
GoVersion:"go1.10.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10",
GitVersion:"v1.10.4",
GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d",
GitTreeState:"clean", BuildDate:"2018-06-06T08:00:59Z",
GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
pip list
|Package | Version|
|certifi | 2018.8.24|
|kubernetes | 7.0.0|
......ommit other dependencies
P.S. I tried the solution here: http://www.cdotson.com/2017/01/sslerror-with-python-3-6-x-on-macos-sierra/. But without luck.
This appears to be an issue with the client script not being able to find/accept/verify the cert. Setting the location of the cert is likely the most viable method of resolution directly from the script:
from kubernetes import client
from kubernetes.client import Configuration, ApiClient
config = Configuration()
config.api_key = {'authorization': 'Bearer <api_key>'}
config.host = 'https://my-kubernetes-cluster'
config.ssl_ca_cert = "/path/to/ca_chain.crt"
api_client = ApiClient(configuration=config)
v1 = client.CoreV1Api(api_client)
v1.list_pod_for_all_namespaces(watch=False)
You could also set this up directly in .kube/config:
apiVersion: v1
clusters:
- cluster:
api-version: v1
certificate-authority: /path/to/ca_chain.crt
server: "https://my-kubernetes-cluster"
...
The last option is to skip verification ( not recommend because it defeats the purpose of SSL ):
v1 = client.CoreV1Api()
v1.api_client.configuration.verify_ssl = False
More Information:
• https://github.com/kubernetes-client/python/issues/521
• How to specify a CA bundle in kubernetes python client
My server on Heroku [seemingly] randomly returns H18 errors.
I use:
Gunicorn (I tried both sync and gevent workers)
Flask
Flask-Restful
If I repeat the same request several times, I occasionally get an H18. I noticed that the most often I get such a response when an exception is raised in my code. I have a handler defined that returns a valid HTTP error response and most of the time it succeeds. Sometimes it fails though - for the reason that I cannot understand.
Here's how the "Bad situation" log looks like:
2018-03-10T17:50:51.875115+00:00 app[web.1]: [2018-03-10 17:50:51 +0000] [11] [DEBUG] PUT /api/v1/views/270
2018-03-10T17:50:51.879035+00:00 app[web.1]: PPMException: Token expired
2018-03-10T17:50:51.879442+00:00 app[web.1]: Traceback (most recent call last):
2018-03-10T17:50:51.879445+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
2018-03-10T17:50:51.879447+00:00 app[web.1]: rv = self.dispatch_request()
2018-03-10T17:50:51.879449+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
2018-03-10T17:50:51.879451+00:00 app[web.1]: return self.view_functions[rule.endpoint](**req.view_args)
2018-03-10T17:50:51.879452+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask_restful/__init__.py", line 480, in wrapper
2018-03-10T17:50:51.879456+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/views.py", line 84, in view
2018-03-10T17:50:51.879454+00:00 app[web.1]: resp = resource(*args, **kwargs)
2018-03-10T17:50:51.879458+00:00 app[web.1]: return self.dispatch_request(*args, **kwargs)
2018-03-10T17:50:51.879464+00:00 app[web.1]: resp = meth(*args, **kwargs)
2018-03-10T17:50:51.879466+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask_httpauth.py", line 88, in decorated
2018-03-10T17:50:51.879463+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask_restful/__init__.py", line 595, in dispatch_request
2018-03-10T17:50:51.879469+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask_httpauth.py", line 125, in authenticate
2018-03-10T17:50:51.879468+00:00 app[web.1]: if not self.authenticate(auth, password):
2018-03-10T17:50:51.879471+00:00 app[web.1]: return self.verify_password_callback(username, client_password)
2018-03-10T17:50:51.879473+00:00 app[web.1]: File "/app/app/api_security.py", line 31, in verify_password
2018-03-10T17:50:51.879477+00:00 app[web.1]: PPMException: Token expired
2018-03-10T17:50:51.879475+00:00 app[web.1]: raise PPMException("SECURITY_TOKEN_EXPIRED", http_code=401)
2018-03-10T17:50:51.879479+00:00 app[web.1]:
2018-03-10T17:50:51.881229+00:00 app[web.1]: 10.189.169.232 - eyJleHAiOjE1MjA3MDQyNTAsImlhdCI6MTUyMDcwNDI0MCwiYWxnIjoiSFMyNTYifQ.eyJ1c2VyX2lkIjoiMyIsIm9yaWdpbmFsX2lzc3VlX2RhdGUiOiIyMDE4LTAzLTEwVDE3OjQ5OjI0Ljk3NTkwMCJ9.mB_NCtrGeDSq0ATs-Zl-Fg8qYRCx8YyIkjVdaLi_O6c
[10/Mar/2018:17:50:51 +0000] "PUT /api/v1/views/270 HTTP/1.1" 401 91 "http://localhost:3000/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
2018-03-10T17:50:51.881448+00:00 app[web.1]: [2018-03-10 17:50:51 +0000] [11] [DEBUG] Closing connection.
2018-03-10T17:50:51.975825+00:00 heroku[router]: sock=backend at=error code=H18 desc="Server Request Interrupted" method=PUT path="/api/v1/views/270" host=***-dev.herokuapp.com request_id=68b4ddb1-19b4-4753-9b3f-395c80fc777a fwd="81.164.187.197" dyno=web.1 connect=2ms service=106ms status=503 bytes=483 protocol=https
vs. how the same exception looks without H18:
2018-03-10T17:53:22.660154+00:00 app[web.1]: [2018-03-10 17:53:22 +0000] [10] [DEBUG] PUT /api/v1/views/270
2018-03-10T17:53:22.667461+00:00 app[web.1]: PPMException: Token expired
2018-03-10T17:53:22.669037+00:00 app[web.1]: Traceback (most recent call last):
2018-03-10T17:53:22.669041+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
2018-03-10T17:53:22.669042+00:00 app[web.1]: rv = self.dispatch_request()
2018-03-10T17:53:22.669044+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
2018-03-10T17:53:22.669046+00:00 app[web.1]: return self.view_functions[rule.endpoint](**req.view_args)
2018-03-10T17:53:22.669048+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask_restful/__init__.py", line 480, in wrapper
2018-03-10T17:53:22.669050+00:00 app[web.1]: resp = resource(*args, **kwargs)
2018-03-10T17:53:22.669055+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/views.py", line 84, in view
2018-03-10T17:53:22.669057+00:00 app[web.1]: return self.dispatch_request(*args, **kwargs)
2018-03-10T17:53:22.669059+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask_restful/__init__.py", line 595, in dispatch_request
2018-03-10T17:53:22.669060+00:00 app[web.1]: resp = meth(*args, **kwargs)
2018-03-10T17:53:22.669062+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask_httpauth.py", line 88, in decorated
2018-03-10T17:53:22.669064+00:00 app[web.1]: if not self.authenticate(auth, password):
2018-03-10T17:53:22.669070+00:00 app[web.1]: File "/app/app/api_security.py", line 31, in verify_password
2018-03-10T17:53:22.669075+00:00 app[web.1]:
2018-03-10T17:53:22.669066+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask_httpauth.py", line 125, in authenticate
2018-03-10T17:53:22.669068+00:00 app[web.1]: return self.verify_password_callback(username, client_password)
2018-03-10T17:53:22.669071+00:00 app[web.1]: raise PPMException("SECURITY_TOKEN_EXPIRED", http_code=401)
2018-03-10T17:53:22.669073+00:00 app[web.1]: PPMException: Token expired
2018-03-10T17:53:22.677909+00:00 app[web.1]: 10.31.61.151 - eyJleHAiOjE1MjA3MDQyNTAsImlhdCI6MTUyMDcwNDI0MCwiYWxnIjoiSFMyNTYifQ.eyJ1c2VyX2lkIjoiMyIsIm9yaWdpbmFsX2lzc3VlX2RhdGUiOiIyMDE4LTAzLTEwVDE3OjQ5OjI0Ljk3NTkwMCJ9.mB_NCtrGeDSq0ATs-Zl-Fg8qYRCx8YyIkjVdaLi_O6c [10/Mar/2018:17:53:22 +0000] "PUT /api/v1/views/270 HTTP/1.1" 401 91 "http://localhost:3000/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
***2018-03-10T17:53:22.678478+00:00 heroku[router]: at=info method=PUT path="/api/v1/views/270" host=***-dev.herokuapp.com request_id=9e22f6a9-c980-478f-9ff1-bc8bbd4fe8c7 fwd="81.164.187.197" dyno=web.1 connect=0ms service=19ms status=401 bytes=483 protocol=https
2018-03-10T17:53:22.678989+00:00 app[web.1]: [2018-03-10 17:53:22 +0000] [10] [DEBUG] Closing connection.
The only difference I see is that in the first case I first see Gunicorn saying "Closing connection" and then the heroku router saying H18, while in the second case heroku router reports the response and then Gunicorn is closing the connection. If Gunicorn is closing connection prematurely, is there a way for me to control it from the application?
I think I got this figured out.
The reason seems to be that there were still request data left unread, so I added "request.stream.read()" in my error handling code.
I am deploy local instance application readthedocs. When I'm start the generation of documentation appear errors.
Here logs:
29/May/2015 08:47:00] INFO [requests.packages.urllib3.connectionpool:180] Starting new HTTP connection (1): localhost
[29/May/2015 08:47:00] INFO [projects.tasks:87] (Build) [diclonius-doc:] Building
[29/May/2015 08:47:00] ERROR [projects.tasks:576] Unable to post a new build
Traceback (most recent call last):
File "/srv/readthedoc/readthedocs/projects/tasks.py", line 574, in record_build
api.build(build['id']).put(build)
File "/srv/readthedoc/.env/local/lib/python2.7/site-packages/slumber/__init__.py", line 164, in put
resp = self._request("PUT", data=data, files=files, params=kwargs)
File "/srv/readthedoc/.env/local/lib/python2.7/site-packages/slumber/__init__.py", line 108, in _request
raise exceptions.HttpClientError("Client Error %s: %s" % (resp.status_code, url), response=resp, content=resp.content)
HttpClientError: Client Error 401: http://localhost:8000/api/v1/build/7/
[29/May/2015 08:47:00] INFO [projects.tasks:209] (Build) [diclonius-doc:latest] Updating docs from VCS
...
...
[29/May/2015 08:47:01] INFO [vcs_support.base:66] VCS[diclonius-doc:latest]: 47973225b291c752663da75d5ba218e5423029c4
[29/May/2015 08:47:01] ERROR [projects.tasks:576] Unable to post a new build
Traceback (most recent call last):
File "/srv/readthedoc/readthedocs/projects/tasks.py", line 574, in record_build
api.build(build['id']).put(build)
File "/srv/readthedoc/.env/local/lib/python2.7/site-packages/slumber/__init__.py", line 164, in put
resp = self._request("PUT", data=data, files=files, params=kwargs)
File "/srv/readthedoc/.env/local/lib/python2.7/site-packages/slumber/__init__.py", line 108, in _request
raise exceptions.HttpClientError("Client Error %s: %s" % (resp.status_code, url), response=resp, content=resp.content)
HttpClientError: Client Error 401: http://localhost:8000/api/v1/build/7/
[29/May/2015 08:47:01] INFO [projects.tasks:308] (Build) [diclonius-doc:latest] Removing existing build dir
[29/May/2015 08:47:01] INFO [projects.utils:81] Running: 'virtualenv-2.7 -p python --no-site-packages /srv/readthedoc/user_builds/diclonius-doc/envs/latest' [/srv/readthedoc]
[29/May/2015 08:47:02] INFO [projects.utils:81] Running: '/srv/readthedoc/user_builds/diclonius-doc/envs/latest/bin/pip install --use-wheel --find-links=/srv/readthedoc/deploy/wheels -U sphinx==1.3.1 Pygments==2.0.2 virtualenv==1.10.1 setuptools==1.1 docutils==0.11 mkdocs==0.11.1 mock==1.0.1 pillow==2.6.1 readthedocs-sphinx-ext==0.4.4 sphinx-rtd-theme==0.1.8 recommonmark==0.1.1' [/srv/readthedoc]
I am use server gunicorn and nginx for proxy.