elasticsearch can not rebuild_index - elasticsearch

I want using haystack + elasticsearch
so I install elasticsearch
$ brew info elasticsearch
elasticsearch: stable 5.0.1, HEAD
Distributed search & analytics engine
https://www.elastic.co/products/elasticsearch
/usr/local/Cellar/elasticsearch/5.0.1 (98 files, 34.8M) *
Built from source on 2016-11-29 at 17:52:15
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/elasticsearch.rb
==> Requirements
Required: java >= 1.8 ✔
==> Caveats
Data: /usr/local/var/elasticsearch/elasticsearch_hanminsoo/
Logs: /usr/local/var/log/elasticsearch/elasticsearch_hanminsoo.log
Plugins: /usr/local/Cellar/elasticsearch/5.0.1/libexec/plugins/
Config: /usr/local/etc/elasticsearch/
plugin script: /usr/local/Cellar/elasticsearch/5.0.1/libexec/bin/plugin
To have launchd start elasticsearch now and restart at login:
brew services start elasticsearch
Or, if you don't want/need a background service you can just run:
elasticsearch
and I start elasticsearch
$ brew services start elasticsearch
==> Tapping homebrew/services
Cloning into '/usr/local/Homebrew/Library/Taps/homebrew/homebrew-services'...
remote: Counting objects: 10, done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 10 (delta 0), reused 6 (delta 0), pack-reused 0
Unpacking objects: 100% (10/10), done.
Checking connectivity... done.
Tapped 0 formulae (36 files, 47K)
==> Successfully started `elasticsearch` (label: homebrew.mxcl.elasticsearch)
and I want rebulid my index
$ python manage.py rebuild_index
WARNING: This will irreparably remove EVERYTHING from your search index in connection 'default'.
Your choices after this are to restore from backups or rebuild via the `rebuild_index` command.
Are you sure you wish to continue? [y/N] y
but It show error
Removing all documents from your index because you said so.
DELETE http://127.0.0.1:9200/haystack [status:N/A request:0.002s
[...]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/hanminsoo/.pyenv/versions/spec/lib/python3.5/site-packages/haystack/backends/elasticsearch_backend.py", line 231, in clear
self.conn.indices.delete(index=self.index_name, ignore=404)
File "/Users/hanminsoo/.pyenv/versions/spec/lib/python3.5/site-packages/elasticsearch/client/utils.py", line 69, in _wrapped
return func(*args, params=params, **kwargs)
File "/Users/hanminsoo/.pyenv/versions/spec/lib/python3.5/site-packages/elasticsearch/client/indices.py", line 198, in delete
params=params)
File "/Users/hanminsoo/.pyenv/versions/spec/lib/python3.5/site-packages/elasticsearch/transport.py", line 307, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/Users/hanminsoo/.pyenv/versions/spec/lib/python3.5/site-packages/elasticsearch/connection/http_urllib3.py", line 89, in perform_request
raise ConnectionError('N/A', str(e), e)
elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x110791e48>: Failed to establish a new connection: [Errno 61] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x110791e48>: Failed to establish a new connection: [Errno 61] Connection refused)
All documents removed.
Indexing 0 products
the products doesn't indexing...
first I think it is connection error,
$ curl -v -X GET 127.0.0.1:9200
* Rebuilt URL to: 127.0.0.1:9200/
* Trying 127.0.0.1...
* connect to 127.0.0.1 port 9200 failed: Connection refused
* Failed to connect to 127.0.0.1 port 9200: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 127.0.0.1 port 9200: Connection refused
but I settings settings.py
# Django Haystack
HAYSTACK_CONNECTIONS = {
'default': {
'ENGINE':'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
'URL': 'http://127.0.0.1:9200/',
'INDEX_NAME': 'haystack',
'TIMEOUT': 10
},
}
I try elasticsearch restart, and pip reinstall, else...
but I can not understand why.....
pip freeze
appnope==0.1.0
certifi==2016.9.26
decorator==4.0.10
Django==1.9.7
django-debug-toolbar==1.5
django-extensions==1.7.2
django-haystack==2.5.1
djangorestframework==3.4.4
elasticsearch==1.9.0
get==0.0.0
httpie==0.9.6
ipython==5.0.0
ipython-genutils==0.1.0
pep8==1.7.0
pexpect==4.2.0
pickleshare==0.7.3
post==0.0.0
prompt-toolkit==1.0.3
ptyprocess==0.5.1
public==0.0.0
pyelasticsearch==1.4
Pygments==2.1.3
query-string==0.0.0
requests==2.12.2
setupfiles==0.0.0
simplegeneric==0.8.1
simplejson==3.10.0
six==1.10.0
sqlparse==0.2.1
traitlets==4.2.2
urllib3==1.19.1
wcwidth==0.1.7
Java -verions
java version "1.8.0_112"
Java(TM) SE Runtime Environment (build 1.8.0_112-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.112-b16, mixed mode)

It's because your ES server is not starting.
According to your logs, you need to change discovery.zen.ping.timeout to discovery.zen.ping_timeout in your /usr/local/etc/elasticsearch/elasticsearch.yml file and restart ES.

Related

After upgrading Ansible to 2.9.9 cannot connect to Windows Hosts, hostname has disappeared from Ansible logs

I recently upgraded Ansible from 2.5.3 to 2.9.9. OS is RHEL 7.5, Python 2.7.
Windows module setup.ps1 cannot fetch the host name of target windows host and hence the playbook unable to connect windows VM, after upgrading to Ansible 2.9.9.
before ESTABLISH WINRM connection log statement you can see that
<> is empty. Hence ansible windows connection has stopped working.
Using module file /usr/lib/python2.7/site-packages/ansible/modules/windows/setup.ps1
Pipelining is enabled.
**<>** ESTABLISH WINRM CONNECTION FOR USER: 7771234s_tskSV on PORT 5985 TO
creating Kerberos CC at /tmp/tmpn5jvwl
calling kinit with subprocess for principal 7771234s_tskSV
Using module file /usr/lib/python2.7/site-packages/ansible/modules/windows/setup.ps1
Pipelining is enabled.
<> ESTABLISH WINRM CONNECTION FOR USER: 7771234s_tskSV on PORT 5985 TO
creating Kerberos CC at /tmp/tmp0hJgs1
calling kinit with subprocess for principal 7771234s_tskSV
kinit succeeded for principal 7771234s_tskSV
**<> WINRM CONNECT: transport=kerberos endpoint=http://:5985/wsman
<> WINRM CONNECTION ERROR: Invalid URL u'http://:5985/wsman': No host supplied what to do**
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/winrm.py", line 415, in _winrm_connect
self.shell_id = protocol.open_shell(codepage=65001) # UTF-8
File "/usr/lib/python2.7/site-packages/winrm/protocol.py", line 157, in open_shell
res = self.send_message(xmltodict.unparse(req))
File "/usr/lib/python2.7/site-packages/winrm/protocol.py", line 234, in send_message
resp = self.transport.send_message(message)
File "/usr/lib/python2.7/site-packages/winrm/transport.py", line 243, in send_message
self.build_session()
File "/usr/lib/python2.7/site-packages/winrm/transport.py", line 232, in build_session
self.setup_encryption()
File "/usr/lib/python2.7/site-packages/winrm/transport.py", line 237, in setup_encryption
prepared_request = self.session.prepare_request(request)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 437, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "/usr/lib/python2.7/site-packages/requests/models.py", line 305, in prepare
self.prepare_url(url, params)
File "/usr/lib/python2.7/site-packages/requests/models.py", line 382, in prepare_url
raise InvalidURL("Invalid URL %r: No host supplied what to do" % url)
InvalidURL: Invalid URL u'http://:5985/wsman': No host supplied what to do
Can you please suggest if the setup.ps1 is not up to date?

Heroku + Redis Cloud: Temporary failure in name resolution

I am using the Redis Cloud add-in on my Heroku application and I keep getting this error sporadically. I have tried flushing the redis DB and restarting dynos and that seems to fix it but I am curious why this is happening so often.
I am running worker dynos that use this redis DB and I am using python-rq to schedule jobs on the worker queues.
File "/usr/local/lib/python3.8/site-packages/redis/connection.py", line 563, in connect
raise ConnectionError(self._error_message(e))
2020-08-05T17:12:25.451733+00:00 app[worker_proc.5]: redis.exceptions.ConnectionError: Error -3 connecting to redis-13618.c73.us-east-1-2.ec2.cloud.redislabs.com:13618. Temporary failure in name resolution.
2020-08-05 17:12:25,461 INFO exited: worker_proc-0 (exit status 1; not expected)
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/redis/connection.py", line 559, in connect
sock = self._connect()
File "/usr/local/lib/python3.8/site-packages/redis/connection.py", line 584, in _connect
for res in socket.getaddrinfo(self.host, self.port, self.socket_type,
File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

Cloudera Manager - Service Monitor is not running

It seems there is something wrong with connection issues.But I don't know what lead to this problem.Could anyone help me?What connection issue am I facing?
(9 skipped) Error sending messages to firehose: mgmt-SERVICEMONITOR-73cd125b24d5013535b3d65d46703b65
Traceback (most recent call last):
File "/opt/cm-5.13.0/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.13.0-py2.6.egg/cmf/monitor/firehose.py", line 116, in _send
self._port)
File "/opt/cm-5.13.0/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 469, in __init__
self.conn.connect()
File "/usr/lib64/python2.6/httplib.py", line 720, in connect
self.timeout)
File "/usr/lib64/python2.6/socket.py", line 567, in create_connection
raise error, msg
error: [Errno 111] Connection refused
The error message indicates that CM agent is not able to reach out to Service Monitor due to Connection refused.
The most common reason is Service Monitor not running. Check if Service Monitor is actually running.
Another reason could be CM agent is not able to connect Service Monitor on port 9997. Check from CM agent host using
telnet <SMON_Host> 9997

HAWQ stop cluster failed

I installed HAWQ from source code. After initializing and starting HAWQ cluster, I tried to stop it with "hawq stop cluster". However, it failed.
The error shows:
[hadoop#Master ~]$ hawq stop cluster
20161217:19:59:31:004594 hawq_stop:Master:hadoop-[INFO]:-Prepare to do 'hawq stop'
20161217:19:59:31:004594 hawq_stop:Master:hadoop-[INFO]:-You can check log in /home/hadoop/hawqAdminLogs/hawq_stop_20161217.log
20161217:19:59:31:004594 hawq_stop:Master:hadoop-[INFO]:-Stop hawq with args: ['stop', 'cluster']
Continue with HAWQ service stop Yy|Nn (default=N):
20161217:19:59:38:004594 hawq_stop:Master:hadoop-[INFO]:-No standby host configured
20161217:19:59:38:004594 hawq_stop:Master:hadoop-[INFO]:-Stop hawq cluster
Traceback (most recent call last):
File "/home/hadoop/hawq/bin/hawq_ctl", line 1276, in <module>
stop_hawq(opts, hawq_dict)
File "/home/hadoop/hawq/bin/hawq_ctl", line 1043, in stop_hawq
instance.run()
File "/home/hadoop/hawq/bin/hawq_ctl", line 891, in run
check_return_code(self._stopAll())
File "/home/hadoop/hawq/bin/hawq_ctl", line 816, in _stopAll
master_result = self._stop_master()
File "/home/hadoop/hawq/bin/hawq_ctl", line 760, in _stop_master
self._stop_master_checks()
File "/home/hadoop/hawq/bin/hawq_ctl", line 712, in _stop_master_checks
self.conn = dbconn.connect(self.dburl, utility=True)
File "/home/hadoop/hawq/lib/python/gppylib/db/dbconn.py", line 211, in connect
cnx = pgdb._connect_(cstr, dbhost, dbport, dbopt, dbtty, dbuser, dbpasswd)
AttributeError: 'module' object has no attribute '_connect_'
At present, I used the alternative way to stop the cluster, that is, stop master and segments separately with pg_ctl.
pg_ctl stop -D <master_data_dir>/<segment_data_dir>
Anything about this error is helpful. Thanks!
Because directly use the command 'pip install pygresql', it will install the latest version(5.0.3) pygresql. In the errors above, pgdb._connect_() is the old version (4.2.2) routine, in 5.0.3 it is pgdb._connect().
The solution is :
pip install pygresql==4.2.2
Before stop cluster, if it's not '-M immediate' stop, hawq will connect to database to check running connections.
From your log, the connection to master node is failed due to python module issues. Seems like pygresql module is not installed properly. Please try to reinstall it.

Scrapyd on Heroku can't recognize rewritten DATABASE_URL by heroku-buildpack-pgbouncer

Okay, here is my setup. I'm on Heroku running a scrapyd daemon using the scrapy-heroku package https://github.com/dmclain/scrapy-heroku.
I'm having issues running out of database connections. I decided to try pooling the database connections use pgbouncer. I'm using this buildpack: https://github.com/heroku/heroku-buildpack-pgbouncer
My procfile was:
web: scrapyd
And I changed it to:
web: bin/start-pgbouncer-stunnel scrapyd
The buildpack is supposed to rewrite your DATABASE_URL when it initializes so that whatever child process is run can just use the DATABASE_URL as normal but will now be connecting to pgbouncer instead of directly to the database.
Within scrapy I'm using adbapi to create a pool for each spider as such:
def from_settings(cls, settings):
dbargs = dict(
host=settings['MYSQL_HOST'],
database=settings['MYSQL_DBNAME'],
user=settings['MYSQL_USER'],
password=settings['MYSQL_PASSWD'],
#charset='utf8',
#use_unicode=True,
)
dbpool = adbapi.ConnectionPool('psycopg2', cp_max=2, cp_min=1, **dbargs)
return cls(dbpool)
And in my settings this is how I'm getting the DATABASE_URL info:
import urlparse
urlparse.uses_netloc.append("postgres")
url = urlparse.urlparse(os.environ["DATABASE_URL"])
MYSQL_HOST = url.hostname
MYSQL_DBNAME = url.path[1:]
MYSQL_USER = url.username
MYSQL_PASSWD = url.password
This was working fine before I added pgbouncer buildpack. Now I get connection errors:
Traceback (most recent call last):
File "/app/.heroku/python/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
result = f(*args, **kw)
File "/app/.heroku/python/lib/python2.7/site-packages/scrapy/xlib/pydispatch/robustapply.py", line 57, in robustApply
return receiver(*arguments, **named)
File "/tmp/etc/etc/etc/middlewares.py", line 92, in spider_opened
File "/app/.heroku/python/lib/python2.7/site-packages/psycopg2/__init__.py", line 164, in connect
conn = _connect(dsn, connection_factory=connection_factory, async=async)
OperationalError: could not connect to server: Connection refused
Is the server running on host "127.0.0.1" and accepting
TCP/IP connections on port 5432?
Does anyone have an idea what the issue may be?

Resources