Is there a way to set a timeout in psycopg2 for db transactions or for db queries?
A sample use-case:
Heroku limits django web requests to 30sec, after which Heroku terminates the request without allowing django to gracefully roll-back any transactions which have not yet returned. This can leave outstanding transactions open on postgres. You could configure a timeout in the database, but that would also limit non-web-related queries such as maintenance scripts analytics etc. In this case setting a timeout via the middleware (or via django) would be preferable.
You can set the timeout at connection time using the options parameter. The syntax is a bit weird:
>>> import psycopg2
>>> cnn = psycopg2.connect("dbname=test options='-c statement_timeout=1000'")
>>> cur = cnn.cursor()
>>> cur.execute("select pg_sleep(2000)")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
psycopg2.extensions.QueryCanceledError: canceling statement due to statement timeout
it can also be set using an env variable:
>>> import os
>>> os.environ['PGOPTIONS'] = '-c statement_timeout=1000'
>>> import psycopg2
>>> cnn = psycopg2.connect("dbname=test")
>>> cur = cnn.cursor()
>>> cur.execute("select pg_sleep(2000)")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
psycopg2.extensions.QueryCanceledError: canceling statement due to statement timeout
You can set a per-statement timeout at any time using SQL. For example:
SET statement_timeout = '2s'
will abort any statement (following it) that takes more than 2 seconds (you can use any valid unit as 's' or 'ms'). Note that when a statement timeouts, psycopg raises an exception and it is your care to catch it and act appropriately.
Looks like PostgreSQL 9.6 added idle transaction timeouts. See:
https://www.postgresql.org/docs/9.6/static/runtime-config-client.html#GUC-IDLE-IN-TRANSACTION-SESSION-TIMEOUT for reference.
http://blog.dbi-services.com/a-look-at-postgresql-9-6-killing-idle-transactions-automatically/ as an example.
PostgreSQL 9.6 is also supported in Heroku so you should be able to use this.
Related
I'm trying to stablish connection to Heroku Postgres Database and a receive this error:
psycopg2.OperationalError:
FATAL: password authentication failed for user "hdyarfoicbluoo"
FATAL: no pg_hba.conf entry for host "95.92.208.27", user "hdyarfoicbluoo", database "d7jcaupbs6m4ud", no encryption
My code is:
import psycopg2
conn = psycopg2.connect( dbname=DB_NAME, user=DB_USER, password=DB_PASS, host=DB_HOST)
conn.close()
I look forward to receiving an answer.
I tried to use DATABASE_URL but don't work, either:
conn = psycopg2.connect(DATABASE_URL, sslmode='require')
Traceback (most recent call last): File "/home/aluno-di/Desktop/dbaccess.py", line 9, in <module>
DATABASE_URL = os.environ['postgres://hdyarfoicbluoo:xxxx#ec2-63-32-248-14.eu-west-1.compute.amazonaws.com:5432/d7jcaupbs6m4ud'] File "/usr/lib/python3.8/os.py", line 675, in __getitem__ raise KeyError(key) from None KeyError: 'postgres://hdyarfoicbluoo:xxxxx#ec2-63-32-248-14.eu-west-1.compute.amazonaws.com:5432/d7jcaupbs6m4ud'
In the first case you are missing the sslmode argument:
conn = psycopg2.connect(
dbname=DB_NAME,
user=DB_USER,
password=DB_PASS,
host=DB_HOST,
sslmode="require", # <-- Here
)
But it is better to use the DATABASE_URL environment variable, as you try to do in your second example. Heroku may change your database credentials at any time, and always using this variable is the best way to ensure that your application continues to work.
The whole point of storing connection strings in environment variables is so you don't have to put them directly in your code. Use the name of the environment variable to retrieve its value.
Something like this should work:
DATABASE_URL = os.environ["DATABASE_URL"]
conn = psycopg2.connect(DATABASE_URL)
I'm writing some middle-ware in Python that uses Socket Mode. Non-async testing works fine. But the connect() call fails:
app_token='xapp-XXX'
bot_token='xoxb-XXX'
web_client = AsyncWebClient(bot_token)
sm_client = SocketModeClient(app_token=app_token, web_client=web_client)
sm_client.connect()
Returns:
Traceback (most recent call last):
File "wrapper.py", line 57, in <module>
sm_client.connect()
File "/Library/Python/3.8/site-packages/slack_sdk/socket_mode/builtin/client.py", line 132, in connect
self.wss_uri = self.issue_new_wss_url()
File "/Library/Python/3.8/site-packages/slack_sdk/socket_mode/client.py", line 48, in issue_new_wss_url
return response["url"]
TypeError: 'coroutine' object is not subscriptable
sys:1: RuntimeWarning: coroutine 'AsyncWebClient.apps_connections_open' was never awaited
I can't find anything in the API or SDK documentation about how to use AsyncWebClient in a Socket Mode application. Is this possible?
In your traceback, it is referencing the builtin client, aka the concurrent client. You're probably importing this:
from slack_sdk.socket_mode import SocketModeClient
You need to use the async client if you're using the AsyncWebClient:
from slack_sdk.socket_mode.aiohttp import SocketModeClient
I have a Flask app running on nginx + uWSGI.
On my local server (non-nginx), I get a nice stack trace + error reporting for exceptions.
Like this:
$ python run.py
Traceback (most recent call last):
File "run.py", line 1, in <module>
from myappname import app
File "/home/me/myappname/myappname/__init__.py", line 27, in <module>
file_handler.setLevel(logging.debug)
File "/usr/lib/python2.7/logging/__init__.py", line 710, in setLevel
self.level = _checkLevel(level)
File "/usr/lib/python2.7/logging/__init__.py", line 190, in _checkLevel
raise TypeError("Level not an integer or a valid string: %r" % level)
On nginx, there is next to no logging whatsoever (in /var/log/nginx/error.log).
This post suggests adding app.logger.exception('Failed') to my script, which didn't help.
How do I enable this sort of logging for debugging purposes?
Nginx will capture your app's console output, but you must make the app recover from exceptions. Else, you'll only get 500 or 400 errors from Nginx.
Try running the app off Nginx until it seems stable.
Use the logging module to capture app status information to your own log file. This strategy will be useful in the long run.
I have installed Celery 3.1.5, RabbitMQ server 3.2.1 and Python 2.7.5 on Windows 7 64 bit machine. Here is my code which copied from first-steps-with-celery.
from celery import Celery
app = Celery('tasks', backend='amqp', broker='amqp://guest#localhost//')
#app.task
def add(x, y):
return x + y
When I execute task from python shell I got "The operation timed out" exception message. And state and ready() always returns PENDING & False.
>>> from tasks import *
>>> result = add.delay(4, 4)
>>> result.ready()
False
>>> result.state
'PENDING'
>>> result.get(timeout=20)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\site-packages\celery\result.py", line 136, in get
interval=interval)
File "C:\Python27\lib\site-packages\celery\backends\amqp.py", line 154, in wait_for
raise TimeoutError('The operation timed out.')
celery.exceptions.TimeoutError: The operation timed out.
>>>
I verified RabbitMQ server is running however I have no clue why celery throwing exception.
You can try to start the worker with command
celery -A proj worker -l info --pool==solo
Though there are lots of things that can cause the result.get() call to fail -- because there are a lot of steps in the chain between sending the message via the .delay() command, to Celery, to the broker (RabbitMQ), and back to a Celery worker, which does the work, and posts the results back, etc. -- I had this problem and the solution was the one that #Deja_vu suggested of "--pool=solo" (note one equals sign, not two).
The default "pool" option is "prefork" (see http://docs.celeryproject.org/en/latest/reference/celery.bin.worker.html#module-celery.bin.worker ). So this may be a Celery bug in its "prefork" system under Windows: see https://github.com/celery/celery/issues/2146
Related StackOverflow questions:
Celery 'Getting Started' not able to retrieve results; always pending
Trouble getting result from Celery queue
In python, I am using the following:
context = zmq.Context()
socket = context.socket(zmq.PUSH)
socket.bind_to_random_port('tcp://*', min_port=6001, max_port=6004, max_tries=100)
port_selected = socket.???????
How do I know what port is chosen? I will have a look up table in redis for the workers to read.
I am using a push pull model. I need to let workers know what ports to connect to.
I have to do this because I am using the gevent loop in uwsgi and specifying a a plain blind thows and error becuase of a fork. If a use bind_to_random_port then a port is seleced, I just dont know which.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gevent-1.0b2-py2.7-linux-x86_64.egg/gevent/greenlet.py",
line 328, in run
result = self._run(*self.args, **self.kwargs)
File "/home/ubuntu/workspace/rtbopsConfig/rtbServers/rtbUwsgiPixelServer/uwsgiPixelServer.py",
line 43, in sendthis
socket.send(push)
File "/usr/local/lib/python2.7/dist-packages/zmq/green/core.py",
line 173, in send
self._wait_write()
File "/usr/local/lib/python2.7/dist-packages/zmq/green/core.py",
line 108, in _wait_write
assert self.__writable.ready(), "Only one greenlet can be waiting
on this event"
AssertionError: Only one greenlet can be waiting on this event
<Greenlet at 0x2d41370: sendthis('2012-07-02 04:05:15')> failed with
AssertionError
port_selected = socket.bind_to_random_port('tcp://*', min_port=6001, max_port=6004, max_tries=100)