I have a 3 server Consul setup with other nodes as clients
2/3 of the Consul servers respond successfully with
root#ewr70:~# consul operator raft list-peers
Node ID Address State Voter RaftProtocol
ewr70 823e8c0e-fea2-be32-4c0d-bd06437f14e1 IP:8300 follower true 3
ewp1 a733c143-4702-30cb-5ff3-6094b108ea06 IP:8300 follower true 3
la452 ad4e15ac-1d73-00b8-cce7-cd5d5a2254a4 IP:8300 leader true 3
root#la452:~# consul operator raft list-peers
Node ID Address State Voter RaftProtocol
ewr70 823e8c0e-fea2-be32-4c0d-bd06437f14e1 IP:8300 follower true 3
ewp1 a733c143-4702-30cb-5ff3-6094b108ea06 IP:8300 follower true 3
la452 ad4e15ac-1d73-00b8-cce7-cd5d5a2254a4 IP:8300 leader true 3
One of them says no cluster leader
root#ewp1:~# consul operator raft list-peers
Error getting peers: Failed to retrieve raft configuration: Unexpected response code: 500 (No cluster leader)
The server failing to list-peers also displays members of the cluster so it acts like it is working except for this cluster leader problem
What I have tried doing:
Restarting consul servers
Creating peers.json on all servers and restarting
consul server join IP
Here are some parts of the consul log output which I think may help understand after executing systemctl restart consul
Nov 10 11:13:33 ewp1 consul[821921]: ==> Starting Consul agent...
Nov 10 11:13:33 ewp1 consul[821921]: Version: 'IPHIDDEN.2'
Nov 10 11:13:33 ewp1 consul[821921]: Build Date: '2022-09-20 20:30:07 +0000 UTC'
Nov 10 11:13:33 ewp1 consul[821921]: Node ID: 'a733c143-4702-30cb-5ff3-6094b108ea06'
Nov 10 11:13:33 ewp1 consul[821921]: Node name: 'ewp1'
Nov 10 11:13:33 ewp1 consul[821921]: Datacenter: 'proxies' (Segment: '<all>')
Nov 10 11:13:33 ewp1 consul[821921]: Server: true (Bootstrap: true)
Nov 10 11:13:33 ewp1 consul[821921]: Client Addr: [IPHIDDEN.IPHIDDEN IPHIDDEN.IPHIDDEN] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 8600)
Nov 10 11:13:33 ewp1 consul[821921]: Cluster Addr: IPHIDDEN.IPHIDDEN (LAN: 8301, WAN: 8302)
Nov 10 11:13:33 ewp1 consul[821921]: Gossip Encryption: true
Nov 10 11:13:33 ewp1 consul[821921]: Auto-Encrypt-TLS: false
Nov 10 11:13:33 ewp1 consul[821921]: HTTPS TLS: Verify Incoming: false, Verify Outgoing: false, Min Version: TLSv1_2
Nov 10 11:13:33 ewp1 consul[821921]: gRPC TLS: Verify Incoming: false, Min Version: TLSv1_2
Nov 10 11:13:33 ewp1 consul[821921]: Internal RPC TLS: Verify Incoming: false, Verify Outgoing: false (Verify Hostname: false), Min Version: TLSv1_2
Nov 10 11:13:33 ewp1 consul[821921]: ==> Log data will now stream in as it occurs:
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [WARN] agent: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [WARN] agent: skipping file /etc/consul.d/consul.hcl.64IPHIDDEN2-09-22#16:35:16~, extension must be .hcl or .json, or config format must be set
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [WARN] agent: bootstrap = true: do not enable unless necessary
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [WARN] agent.auto_config: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [WARN] agent.auto_config: skipping file /etc/consul.d/consul.hcl.64IPHIDDEN2-09-22#16:35:16~, extension must be .hcl or .json, or config format must be set
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent.server.raft: starting restore from snapshot: id=5-368711-1668054243884 last-index=368711 last-term=5 size-in-bytes=28562
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent.server.raft: snapshot restore progress: id=5-368711-1668054243884 last-index=368711 last-term=5 size-in-bytes=28562 read-bytes=28562 percent-complete=IPHIDDEN%
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent.server.raft: restored from snapshot: id=5-368711-1668054243884 last-index=368711 last-term=5 size-in-bytes=28562
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:823e8c0e-fea2-be32-4c0d-bd06437f14e1 Address:IPHIDDEN.IPHIDDEN:8300} {Suffrage:Voter ID:a733c143-4702-30cb-5ff3-6094b108ea06 Address:IPHIDDEN.IPHIDDEN:8300} {Suffrage:Voter ID:ad4e15ac-1d73-00b8-cce7-cd5d5a2254a4 Address:IPHIDDEN.IPHIDDEN:8300}]"
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent.server.raft: entering follower state: follower="Node at IPHIDDEN.IPHIDDEN:8300 [Follower]" leader-address= leader-id=
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent.server.serf.wan: serf: EventMemberJoin: ewp1.proxies IPHIDDEN.IPHIDDEN
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent.server.serf.wan: serf: Attempting re-join to previously known node: hetz-5.hetz: IPHIDDEN.IPHIDDEN:8302
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent.server.serf.lan: serf: EventMemberJoin: ewp1 IPHIDDEN.IPHIDDEN
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent.router: Initializing LAN area manager
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent.server.serf.lan: serf: Attempting re-join to previously known node: securepri9245: IPHIDDEN.IPHIDDEN:8301
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent.server: Adding LAN server: server="ewp1 (Addr: tcp/IPHIDDEN.IPHIDDEN:8300) (DC: proxies)"
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent.server.autopilot: reconciliation now disabled
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [WARN] agent: [core]grpc: addrConn.createTransport failed to connect to {proxies-IPHIDDEN.IPHIDDEN:8300 ewp1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp IPHIDDEN.IPHIDDEN:0->IPHIDDEN.IPHIDDEN:8300: operation was canceled". Reconnecting...
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent.server: Handled event for server in area: event=member-join server=ewp1.proxies area=wan
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent: Started DNS server: address=IPHIDDEN.IPHIDDEN:8600 network=tcp
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent: Started DNS server: address=IPHIDDEN.IPHIDDEN:8600 network=udp
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent: Started DNS server: address=IPHIDDEN.IPHIDDEN:8600 network=udp
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent: Started DNS server: address=IPHIDDEN.IPHIDDEN:8600 network=tcp
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent: Starting server: address=IPHIDDEN.IPHIDDEN:8500 network=tcp protocol=http
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent: Starting server: address=IPHIDDEN.IPHIDDEN:8500 network=tcp protocol=http
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent: started state syncer
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent: Consul agent running!
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere"
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent: Joining cluster...: cluster=LAN
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent: (LAN) joining: lan_addresses=[proxies-3.datacenter.HOSTHIDDEN.net, proxies-1.datacenter.HOSTHIDDEN.net]
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent: Retry join is supported for the following discovery methods: cluster=WAN discovery_methods="aliyun aws azure digitalocean gce k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere"
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent: Joining cluster...: cluster=WAN
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent: (WAN) joining: wan_addresses=[24.services.HOSTHIDDEN.net]
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent.server.serf.wan: serf: EventMemberJoin: 24-IPHIDDENshells IPHIDDEN.IPHIDDEN
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent.server.serf.wan: serf: EventMemberJoin: hetz-new-nomad-pink.hetz IPHIDDEN.IPHIDDEN
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent.server: Handled event for server in area: event=member-join server=24-IPHIDDENshells area=wan
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent.server.serf.wan: serf: EventMemberJoin: hetz-5.hetz IPHIDDEN.IPHIDDEN
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent.server: Handled event for server in area: event=member-join server=hetz-new-nomad-pink.hetz area=wan
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent.server.serf.wan: serf: EventMemberJoin: 24-IPHIDDENshells IPHIDDEN.IPHIDDEN
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [INFO] agent.server: Handled event for server in area: event=member-join server=hetz-5.hetz area=wan
Nov 10 11:13:33 ewp1 consul[821921]: 2022-11-10T19:13:IPHIDDEN+0800 [WARN] agent.server.memberlist.wan: memberlist: Refuting an alive message for 'ewp1.proxies'
Does anyone know what course of action I need to take to get this server behaving?
Related
after upgrading via mc command i get this error when i try to login to the (kind of new) minio console:
Post "https://fqdn.org/": dial tcp 127.0.1.1:443: connect: connection refused
I have a signed and valid SSL Certificate.
Downgrading minio (aka restore Snapshot of VM) solves the problem.
Any ideas?
This is my config:
MINIO_SERVER_URL="https://fqdn.org"
MINIO_ACCESS_KEY="key"
MINIO_VOLUMES="/mnt/hdd2/minio/"
MINIO_OPTS="-C /etc/minio --address :9000 --console-address :9001"
MINIO_SECRET_KEY="minio"
This is my minio startup log:
● minio.service - MinIO
Loaded: loaded (/etc/systemd/system/minio.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2021-11-11 08:41:14 CET; 4min 50s ago
Docs: https://docs.min.io
Process: 3567 ExecStartPre=/bin/bash -c if [ -z "${MINIO_VOLUMES}" ]; then echo "Variable MINIO_VOLUMES not set in /etc/default/minio"; exit 1; fi (code=exited, status=0/SUCCESS)
Main PID: 3568 (minio)
Tasks: 9 (limit: 2351)
Memory: 101.9M
CGroup: /system.slice/minio.service
└─3568 /home/minio/minio server -C /etc/minio --address :9000 --console-address :9001 /mnt/hdd2/minio/
Nov 11 08:41:14 pmit-minio-test systemd[1]: Starting MinIO...
Nov 11 08:41:14 pmit-minio-test systemd[1]: Started MinIO.
Nov 11 08:41:17 pmit-minio-test minio[3568]: WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
Nov 11 08:41:17 pmit-minio-test minio[3568]: Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
Nov 11 08:41:17 pmit-minio-test minio[3568]: API: https://fqdn.org
Nov 11 08:41:17 pmit-minio-test minio[3568]: Console: https://191.164.213.7:9001 https://127.0.0.1:9001
Nov 11 08:41:17 pmit-minio-test minio[3568]: Documentation: https://docs.min.io
Please see the answer here:
https://github.com/minio/minio/issues/13639#issuecomment-966244704
I had to change this line:
MINIO_SERVER_URL="https://fqdn.org:9000"
I have Elasticsearch running on Kubernetes (EKS), with filebeat running as daemonset on Kubernetes.
Now I am trying to get the logs from other EC2 machines (outside of the EKS), so have installed exact version of filebeat on EC2 and configured it to send logs to Elasticsearch running on Kubernetes.
But not able to see any logs in Elasticsearch (Kibana). Here are the logs for filebeat
2019-08-26T18:18:16.005Z INFO instance/beat.go:292 Setup Beat: filebeat; Version: 7.2.1
2019-08-26T18:18:16.005Z INFO [index-management] idxmgmt/std.go:178 Set output.elasticsearch.index to 'filebeat-7.2.1' as ILM is enabled.
2019-08-26T18:18:16.005Z INFO elasticsearch/client.go:166 Elasticsearch url: http://elasticsearch.dev.domain.net:9200
2019-08-26T18:18:16.005Z INFO add_cloud_metadata/add_cloud_metadata.go:351 add_cloud_metadata: hosting provider type detected as aws, metadata={"availability_zone":"us-west-2a","instance":{"id":"i-0185e1d68306f95b4"},"machine":{"type":"t2.medium"},"provider":"aws","region":"us-west-2"}
2019-08-26T18:18:16.005Z INFO [publisher] pipeline/module.go:97 Beat name: dev-web1
2019-08-26T18:18:16.006Z INFO elasticsearch/client.go:166 Elasticsearch url: http://elasticsearch.dev.domain.net:9200
Not much info in the logs.
Then I notice :
root#dev-web1:~# sudo systemctl status filebeat
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2019-08-26 18:18:47 UTC; 18min ago
Docs: https://www.elastic.co/products/beats/filebeat
Main PID: 7768 (filebeat)
CGroup: /system.slice/filebeat.service
└─7768 /usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs
Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.156Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://elasticsear
Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.156Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(http://elastic
Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.156Z INFO [publisher] pipeline/retry.go:189 retryer: send unwait-signal to consumer
Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.157Z INFO [publisher] pipeline/retry.go:191 done
Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.157Z INFO [publisher] pipeline/retry.go:166 retryer: send wait signal to consumer
Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.157Z INFO [publisher] pipeline/retry.go:168 done
Aug 26 18:35:47 dev-web1 filebeat[7768]: 2019-08-26T18:35:47.028Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitori
Aug 26 18:36:17 dev-web1 filebeat[7768]: 2019-08-26T18:36:17.028Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitori
root#dev-web1:~#
But I can't read complete line in above status message.
So I tried :
root#dev-web1:~# curl elasticsearch.dev.domain.net/_cat/health
1566844775 18:39:35 dev-eks-logs green 3 3 48 24 0 0 0 0 - 100.0%
root#dev-web1:~#
which worked but not with port
root#dev-web1:~# curl elasticsearch.dev.domain.net:9200/_cat/health
filebeat has following config
output.elasticsearch:
hosts: ["elasticsearch.dev.domain.net"]
username: "elastic"
password: "changeme"
How can I fix this at filebeat side ?
Telnet Test :
root#dev-web1:~# telnet <ip> 5044
Trying <ip>...
telnet: Unable to connect to remote host: Connection refused
root#dev-web1:~# telnet localhost 5044
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused
root#dev-web1:~#
https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#hosts-option says:
hosts...If no port is specified, 9200 is used.
Adding hosts: ["elasticsearch.dev.domain.net:80"] in the filbeat configuration should resolve the issue.
I think is a problem of network , check A telnet to localhost/IP 5044
I have Kibana and Elasticsearch on separate VMs.
Elasticsearch IP: 10.3.30.209
Kibana IP: 10.3.30.211
I am unable to curl the IP:PORT of Elasticsearch from Kibana:
curl 10.3.30.209:9200
curl: (7) Failed connect to 10.3.30.209:9200; Connection refused
The firewall on Elesticsearch server is listening for port 9200:
firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: ens192
sources:
services: ssh dhcpv6-client
ports: 9200/tcp
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
The Elasticsearch service was at least starting up before I modified the elasticsearch.yml file to enable listening on the elasticsearch port and bind the Kibana IP:
network.host: 10.3.30.211
# Set a custom port for HTTP:
#
# http.port: 9200
http.port: 9200
Error when starting service:
systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2018-06-29 14:06:49 CDT; 1s ago
▽
Docs: http://www.elastic.co
Process: 6382 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_HOME} -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR} -Des.default.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
Process: 6380 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 6382 (code=exited, status=1/FAILURE)
Jun 29 14:06:49 elasticsearch01 elasticsearch[6382]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Jun 29 14:06:49 elasticsearch01 elasticsearch[6382]: at java.lang.Thread.run(Thread.java:748)
Jun 29 14:06:49 elasticsearch01 elasticsearch[6382]: Refer to the log for complete error details.
Jun 29 14:06:49 elasticsearch01 elasticsearch[6382]: [2018-06-29 14:06:49,189][INFO ][node ] [Turac] stopping ...
Jun 29 14:06:49 elasticsearch01 elasticsearch[6382]: [2018-06-29 14:06:49,199][INFO ][node ] [Turac] stopped
Jun 29 14:06:49 elasticsearch01 elasticsearch[6382]: [2018-06-29 14:06:49,200][INFO ][node ] [Turac] closing ...
Jun 29 14:06:49 elasticsearch01 elasticsearch[6382]: [2018-06-29 14:06:49,204][INFO ][node ] [Turac] closed
Jun 29 14:06:49 elasticsearch01 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jun 29 14:06:49 elasticsearch01 systemd[1]: Unit elasticsearch.service entered failed state.
Jun 29 14:06:49 elasticsearch01 systemd[1]: elasticsearch.service failed.
NOTE: I am able to ping the Elasticsearch server from Kibana.
Any help with figuring out why this change breaks the Elasticsearch service and how I might be able to connect my Kibana server to Elasticsearch would be appreciated!
I try to run Consul image on Mac forwarding 8500 port for simple tests.
My command to run the image is:
docker run -it -p 8500:8500 consul agent -server -bootstrap 0.0.0.0
I do not use --net=host since it does not work on Mac so I try to forward 8500.
When I try to telnet from my Mac the connection gets immediately closed:
user$ telnet localhost 8500
Trying ::1...
Connected to localhost.
Escape character is '^]'.
Connection closed by foreign host.
Or when I try to add a new value I get:
consul kv put foo bar
Error! Failed writing data: Put http://127.0.0.1:8500/v1/kv/foo: dial tcp 127.0.0.1:8500: getsockopt: connection refused
What did I miss?
I have just tried what you have posted and seems that the port 8500 is opened
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f4ac8a5233e2 consul "docker-entrypoint..." 2 minutes ago Up 2 minutes 8300-8302/tcp, 8301-8302/udp, 8600/tcp, 8600/udp, 0.0.0.0:8500->8500/tcp sharp_knuth
And I get this:
Trying 0.0.0.0...
Connected to dev-consul
Escape character is '^]'.
Connection closed by foreign host.
However, it is running as you can see from the logs:
==> Starting Consul agent...
==> Consul agent running!
Version: 'v0.9.3'
Node ID: '27998add-58f9-e424-84a0-038db228629f'
Node name: '68bfdf141e7f'
Datacenter: 'dc1' (Segment: '<all>')
Server: true (Bootstrap: false)
Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2017/10/02 20:26:27 [DEBUG] Using random ID "27998add-58f9-e424-84a0-038db228629f" as node ID
2017/10/02 20:26:27 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:127.0.0.1:8300 Address:127.0.0.1:8300}]
2017/10/02 20:26:27 [INFO] raft: Node at 127.0.0.1:8300 [Follower] entering Follower state (Leader: "")
2017/10/02 20:26:27 [INFO] serf: EventMemberJoin: 68bfdf141e7f.dc1 127.0.0.1
2017/10/02 20:26:27 [INFO] serf: EventMemberJoin: 68bfdf141e7f 127.0.0.1
2017/10/02 20:26:27 [INFO] consul: Adding LAN server 68bfdf141e7f (Addr: tcp/127.0.0.1:8300) (DC: dc1)
2017/10/02 20:26:27 [INFO] consul: Handled member-join event for server "68bfdf141e7f.dc1" in area "wan"
2017/10/02 20:26:27 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2017/10/02 20:26:27 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
2017/10/02 20:26:27 [INFO] agent: Started HTTP server on [::]:8500
2017/10/02 20:26:27 [WARN] raft: Heartbeat timeout from "" reached, starting election
2017/10/02 20:26:27 [INFO] raft: Node at 127.0.0.1:8300 [Candidate] entering Candidate state in term 2
2017/10/02 20:26:27 [DEBUG] raft: Votes needed: 1
2017/10/02 20:26:27 [DEBUG] raft: Vote granted from 127.0.0.1:8300 in term 2. Tally: 1
2017/10/02 20:26:27 [INFO] raft: Election won. Tally: 1
2017/10/02 20:26:27 [INFO] raft: Node at 127.0.0.1:8300 [Leader] entering Leader state
2017/10/02 20:26:27 [INFO] consul: cluster leadership acquired
2017/10/02 20:26:27 [DEBUG] consul: Skipping self join check for "68bfdf141e7f" since the cluster is too small
2017/10/02 20:26:27 [INFO] consul: member '68bfdf141e7f' joined, marking health alive
2017/10/02 20:26:27 [INFO] consul: New leader elected: 68bfdf141e7f
2017/10/02 20:26:28 [INFO] agent: Synced node info
2017/10/02 20:27:27 [DEBUG] consul: Skipping self join check for "68bfdf141e7f" since the cluster is too small
2017/10/02 20:27:34 [DEBUG] agent: Node info in sync
Just reinstalled Mongodb on my mac (fresh install of mountain lion 10.8) and now my apps are taking ~3 mins to connect.
I put together a simple node script to test this:
var start = (new Date()).getTime();
var mongoose = require('mongoose');
var db = mongoose.connect('mongodb://localhost/passport-mongox',function(err){
var stop = (new Date()).getTime();
console.log('Took this long: ',(stop-start) / 1000 );
});
Both times were 175.273 and 175.316 seconds.
When I connect to an external, hosted mongodb it connects in less than a second,
Any idea why this would happen? Here is my mongo.log:
Fri Feb 1 12:43:25 [initandlisten] MongoDB starting : pid=2262 port=27017 dbpath=/usr/local/var/mongodb 64-bit host=w
Fri Feb 1 12:43:25 [initandlisten] db version v2.2.2, pdfile version 4.5
Fri Feb 1 12:43:25 [initandlisten] git version: d1b43b61a5308c4ad0679d34b262c5af9d664267
Fri Feb 1 12:43:25 [initandlisten] build info: Darwin bs-osx-106-x86-64-1.local 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386 BOOST_LIB_VERSION=1_49
Fri Feb 1 12:43:25 [initandlisten] options: { bind_ip: "127.0.0.1", config: "/usr/local/etc/mongod.conf", dbpath: "/usr/local/var/mongodb", logappend: "true", logpath: "/usr/local/var/log/mongodb/mongo.log" }
Fri Feb 1 12:43:25 [initandlisten] journal dir=/usr/local/var/mongodb/journal
Fri Feb 1 12:43:25 [initandlisten] recover : no journal files present, no recovery needed
Fri Feb 1 12:43:26 [websvr] admin web console waiting for connections on port 28017
Fri Feb 1 12:43:26 [initandlisten] waiting for connections on port 27017
Fri Feb 1 12:44:05 [initandlisten] connection accepted from 127.0.0.1:52137 #1 (1 connection now open)
Fri Feb 1 12:44:40 [initandlisten] connection accepted from 127.0.0.1:52152 #2 (2 connections now open)
Fri Feb 1 12:45:15 [initandlisten] connection accepted from 127.0.0.1:52201 #3 (3 connections now open)
Fri Feb 1 12:45:50 [initandlisten] connection accepted from 127.0.0.1:52298 #4 (4 connections now open)
Fri Feb 1 12:46:25 [initandlisten] connection accepted from 127.0.0.1:52325 #5 (5 connections now open)
Fri Feb 1 12:51:26 [conn5] end connection 127.0.0.1:52325 (4 connections now open)
Fri Feb 1 12:51:26 [conn3] end connection 127.0.0.1:52201 (4 connections now open)
Fri Feb 1 12:51:26 [conn4] end connection 127.0.0.1:52298 (4 connections now open)
Fri Feb 1 12:51:26 [conn1] end connection 127.0.0.1:52137 (4 connections now open)
Fri Feb 1 12:51:26 [conn2] end connection 127.0.0.1:52152 (4 connections now open)
Answer from mongoose.js
Cause:
The underlying MongoDB driver defaults to looking for IPv6 addresses,
so the most likely cause is that your localhost DNS mapping isn't configured to handle IPv6.
Solution :
Use 127.0.0.1 instead of localhost or use the family option as shown in the connection docs.
mongoose.connect(url, {family:4}, function(err, connection) {
connection.db(your_db_name);
});
So the answer came from #AdamMeghji on twitter.
My hosts file has always looked like this:
127.0.0.1 localhost
127.0.0.1 test.com
127.0.0.1 wes.dev
I switched that to:
127.0.0.1 localhost test.com wes.dev
and connections went back to 0.015 seconds.