I am using the Redis Cloud add-in on my Heroku application and I keep getting this error sporadically. I have tried flushing the redis DB and restarting dynos and that seems to fix it but I am curious why this is happening so often.
I am running worker dynos that use this redis DB and I am using python-rq to schedule jobs on the worker queues.
File "/usr/local/lib/python3.8/site-packages/redis/connection.py", line 563, in connect
raise ConnectionError(self._error_message(e))
2020-08-05T17:12:25.451733+00:00 app[worker_proc.5]: redis.exceptions.ConnectionError: Error -3 connecting to redis-13618.c73.us-east-1-2.ec2.cloud.redislabs.com:13618. Temporary failure in name resolution.
2020-08-05 17:12:25,461 INFO exited: worker_proc-0 (exit status 1; not expected)
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/redis/connection.py", line 559, in connect
sock = self._connect()
File "/usr/local/lib/python3.8/site-packages/redis/connection.py", line 584, in _connect
for res in socket.getaddrinfo(self.host, self.port, self.socket_type,
File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
Related
I’ve rebooted my system and then run all my containers using the vendor/bin/sail up command, the only one that failed to reload was MySQL. The error is the following :
ERROR: for mysql a bytes-like object is required, not 'str'
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/docker/api/client.py", line 261, in _raise_for_status
response.raise_for_status()
File "/usr/lib/python3/dist-packages/requests/models.py", line 940, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.25/containers/afdd1cbf7f45d9b20612bca
f73eef1b0bc1dd631bc6aa3dcfbf630c64e8a3662/start
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/compose/service.py", line 625, in start_container
container.start()
File "/usr/lib/python3/dist-packages/compose/container.py", line 241, in start
return self.client.start(self.id, **options)
File "/usr/lib/python3/dist-packages/docker/utils/decorators.py", line 19, in wrapped
return f(self, resource_id, *args, **kwargs)
File "/usr/lib/python3/dist-packages/docker/api/container.py", line 1095, in start
self._raise_for_status(res)
File "/usr/lib/python3/dist-packages/docker/api/client.py", line 263, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/usr/lib/python3/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("b'Ports are not available: listen tcp 0.0.0.0:3306: bind: An attempt was made
to access a socket in a way forbidden by its access permissions.'")
I’m running this container on ubuntu server 20.04.
It might fix the problem if you provide an absolute path to your nginx/mysql conf file. I haven't tried the solution yet.
It seems there is something wrong with connection issues.But I don't know what lead to this problem.Could anyone help me?What connection issue am I facing?
(9 skipped) Error sending messages to firehose: mgmt-SERVICEMONITOR-73cd125b24d5013535b3d65d46703b65
Traceback (most recent call last):
File "/opt/cm-5.13.0/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.13.0-py2.6.egg/cmf/monitor/firehose.py", line 116, in _send
self._port)
File "/opt/cm-5.13.0/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 469, in __init__
self.conn.connect()
File "/usr/lib64/python2.6/httplib.py", line 720, in connect
self.timeout)
File "/usr/lib64/python2.6/socket.py", line 567, in create_connection
raise error, msg
error: [Errno 111] Connection refused
The error message indicates that CM agent is not able to reach out to Service Monitor due to Connection refused.
The most common reason is Service Monitor not running. Check if Service Monitor is actually running.
Another reason could be CM agent is not able to connect Service Monitor on port 9997. Check from CM agent host using
telnet <SMON_Host> 9997
Okay, here is my setup. I'm on Heroku running a scrapyd daemon using the scrapy-heroku package https://github.com/dmclain/scrapy-heroku.
I'm having issues running out of database connections. I decided to try pooling the database connections use pgbouncer. I'm using this buildpack: https://github.com/heroku/heroku-buildpack-pgbouncer
My procfile was:
web: scrapyd
And I changed it to:
web: bin/start-pgbouncer-stunnel scrapyd
The buildpack is supposed to rewrite your DATABASE_URL when it initializes so that whatever child process is run can just use the DATABASE_URL as normal but will now be connecting to pgbouncer instead of directly to the database.
Within scrapy I'm using adbapi to create a pool for each spider as such:
def from_settings(cls, settings):
dbargs = dict(
host=settings['MYSQL_HOST'],
database=settings['MYSQL_DBNAME'],
user=settings['MYSQL_USER'],
password=settings['MYSQL_PASSWD'],
#charset='utf8',
#use_unicode=True,
)
dbpool = adbapi.ConnectionPool('psycopg2', cp_max=2, cp_min=1, **dbargs)
return cls(dbpool)
And in my settings this is how I'm getting the DATABASE_URL info:
import urlparse
urlparse.uses_netloc.append("postgres")
url = urlparse.urlparse(os.environ["DATABASE_URL"])
MYSQL_HOST = url.hostname
MYSQL_DBNAME = url.path[1:]
MYSQL_USER = url.username
MYSQL_PASSWD = url.password
This was working fine before I added pgbouncer buildpack. Now I get connection errors:
Traceback (most recent call last):
File "/app/.heroku/python/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
result = f(*args, **kw)
File "/app/.heroku/python/lib/python2.7/site-packages/scrapy/xlib/pydispatch/robustapply.py", line 57, in robustApply
return receiver(*arguments, **named)
File "/tmp/etc/etc/etc/middlewares.py", line 92, in spider_opened
File "/app/.heroku/python/lib/python2.7/site-packages/psycopg2/__init__.py", line 164, in connect
conn = _connect(dsn, connection_factory=connection_factory, async=async)
OperationalError: could not connect to server: Connection refused
Is the server running on host "127.0.0.1" and accepting
TCP/IP connections on port 5432?
Does anyone have an idea what the issue may be?
I have installed Celery 3.1.5, RabbitMQ server 3.2.1 and Python 2.7.5 on Windows 7 64 bit machine. Here is my code which copied from first-steps-with-celery.
from celery import Celery
app = Celery('tasks', backend='amqp', broker='amqp://guest#localhost//')
#app.task
def add(x, y):
return x + y
When I execute task from python shell I got "The operation timed out" exception message. And state and ready() always returns PENDING & False.
>>> from tasks import *
>>> result = add.delay(4, 4)
>>> result.ready()
False
>>> result.state
'PENDING'
>>> result.get(timeout=20)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\site-packages\celery\result.py", line 136, in get
interval=interval)
File "C:\Python27\lib\site-packages\celery\backends\amqp.py", line 154, in wait_for
raise TimeoutError('The operation timed out.')
celery.exceptions.TimeoutError: The operation timed out.
>>>
I verified RabbitMQ server is running however I have no clue why celery throwing exception.
You can try to start the worker with command
celery -A proj worker -l info --pool==solo
Though there are lots of things that can cause the result.get() call to fail -- because there are a lot of steps in the chain between sending the message via the .delay() command, to Celery, to the broker (RabbitMQ), and back to a Celery worker, which does the work, and posts the results back, etc. -- I had this problem and the solution was the one that #Deja_vu suggested of "--pool=solo" (note one equals sign, not two).
The default "pool" option is "prefork" (see http://docs.celeryproject.org/en/latest/reference/celery.bin.worker.html#module-celery.bin.worker ). So this may be a Celery bug in its "prefork" system under Windows: see https://github.com/celery/celery/issues/2146
Related StackOverflow questions:
Celery 'Getting Started' not able to retrieve results; always pending
Trouble getting result from Celery queue
In python, I am using the following:
context = zmq.Context()
socket = context.socket(zmq.PUSH)
socket.bind_to_random_port('tcp://*', min_port=6001, max_port=6004, max_tries=100)
port_selected = socket.???????
How do I know what port is chosen? I will have a look up table in redis for the workers to read.
I am using a push pull model. I need to let workers know what ports to connect to.
I have to do this because I am using the gevent loop in uwsgi and specifying a a plain blind thows and error becuase of a fork. If a use bind_to_random_port then a port is seleced, I just dont know which.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gevent-1.0b2-py2.7-linux-x86_64.egg/gevent/greenlet.py",
line 328, in run
result = self._run(*self.args, **self.kwargs)
File "/home/ubuntu/workspace/rtbopsConfig/rtbServers/rtbUwsgiPixelServer/uwsgiPixelServer.py",
line 43, in sendthis
socket.send(push)
File "/usr/local/lib/python2.7/dist-packages/zmq/green/core.py",
line 173, in send
self._wait_write()
File "/usr/local/lib/python2.7/dist-packages/zmq/green/core.py",
line 108, in _wait_write
assert self.__writable.ready(), "Only one greenlet can be waiting
on this event"
AssertionError: Only one greenlet can be waiting on this event
<Greenlet at 0x2d41370: sendthis('2012-07-02 04:05:15')> failed with
AssertionError
port_selected = socket.bind_to_random_port('tcp://*', min_port=6001, max_port=6004, max_tries=100)