Cannot get wildcard certificate with traefik v2 and godaddy - lets-encrypt

I'm trying to set up a wildcard certificate mechanism with traefik v2.2 and GoDaddy. What I want to do is generating a valid certificate for the URLs pattern *.example.org. Here there is my docker-compose:
version: '3.7'
services:
traefik:
image: traefik:v2.2
container_name: traefik
restart: always
env_file:
- .provider.env
# .provider.env contains `GODADDY_API_KEY` and `GODADDY_API_SECRET`
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- ./tls-certificates:/tls-certificates
ports:
# http
- 8080:80
# https
- 443:443
command:
- --api.dashboard=true
- --providers.docker=true
- --providers.docker.exposedbydefault=false
- --providers.docker.network=proxy
- --entrypoints.webinsecure.address=:80
- --entrypoints.websecure.address=:443
# --certificatesresolvers.<name> Certificates resolvers configuration
# ACME V2 supports wildcard certificates.
# Wildcard certificates can only be generated through a DNS-01 challenge.
- --certificatesresolvers.wildcard-godaddy.acme.tlschallenge=true
- --certificatesResolvers.wildcard-godaddy.acme.dnsChallenge.provider=godaddy
- --certificatesResolvers.wildcard-godaddy.acme.dnsChallenge.delayBeforeCheck=0
# Email address used for registration.
- --certificatesresolvers.wildcard-godaddy.acme.email=foo#example.org
# Certificates storage
- --certificatesresolvers.wildcard-godaddy.acme.storage=/tls-certificates/acme.json
networks:
- proxy
labels:
- "traefik.enable=true"
- "traefik.http.routers.traefik.entrypoints=webinsecure"
- "traefik.http.routers.traefik.rule=Host(`traefik.example.org`)"
- "traefik.http.middlewares.traefik-auth.basicauth.users=${DASHBOARD_USERNAME}:${DASHBOARD_PASSWORD}"
- "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
- "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
- "traefik.http.routers.traefik-secure.entrypoints=websecure"
- "traefik.http.routers.traefik-secure.rule=Host(`traefik.example.org`)"
- "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
- "traefik.http.routers.traefik-secure.tls=true"
- "traefik.http.routers.traefik-secure.tls.certresolver=wildcard-godaddy"
- "traefik.http.routers.traefik-secure.tls.domains[0].main=example.org"
- "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.example.org"
- "traefik.http.routers.traefik-secure.service=api#internal"
networks:
proxy:
external: true
In my dns I have an A record * pointing to my ip address.
However when I start the compose I get the following error:
traefik | time="2020-04-15T16:40:50Z" level=debug msg="No default certificate, generating one"
traefik | time="2020-04-15T16:40:51Z" level=debug msg="Looking for provided certificate(s) to validate [\"example.org\" \"*.example.org\"]..." providerName=wildcard-godaddy.acme
traefik | time="2020-04-15T16:40:51Z" level=debug msg="Domains [\"example.org\" \"*.example.org\"] need ACME certificates generation for domains \"example.org,*.example.org\"." providerName=wildcard-godaddy.acme
traefik | time="2020-04-15T16:40:51Z" level=debug msg="Loading ACME certificates [example.org *.example.org]..." providerName=wildcard-godaddy.acme
traefik | time="2020-04-15T16:40:51Z" level=debug msg="Building ACME client..." providerName=wildcard-godaddy.acme
traefik | time="2020-04-15T16:40:51Z" level=debug msg="https://acme-v02.api.letsencrypt.org/directory" providerName=wildcard-godaddy.acme
traefik | time="2020-04-15T16:40:51Z" level=debug msg="Using DNS Challenge provider: godaddy" providerName=wildcard-godaddy.acme
traefik | time="2020-04-15T16:40:51Z" level=debug msg="Using TLS Challenge provider." providerName=wildcard-godaddy.acme
traefik | time="2020-04-15T16:40:51Z" level=debug msg="legolog: [INFO] [example.org, *.example.org] acme: Obtaining bundled SAN certificate"
traefik | time="2020-04-15T16:40:52Z" level=debug msg="legolog: [INFO] [*.example.org] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/id1"
traefik | time="2020-04-15T16:40:52Z" level=debug msg="legolog: [INFO] [example.org] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/id2"
traefik | time="2020-04-15T16:40:52Z" level=debug msg="legolog: [INFO] [*.example.org] acme: use dns-01 solver"
traefik | time="2020-04-15T16:40:52Z" level=debug msg="legolog: [INFO] [example.org] acme: use tls-alpn-01 solver"
traefik | time="2020-04-15T16:40:52Z" level=debug msg="TLS Challenge Present temp certificate for example.org" providerName=acme
traefik | time="2020-04-15T16:40:52Z" level=debug msg="legolog: [INFO] [example.org] acme: Trying to solve TLS-ALPN-01"
traefik | time="2020-04-15T16:40:58Z" level=debug msg="TLS Challenge CleanUp temp certificate for example.org" providerName=acme
traefik | time="2020-04-15T16:40:58Z" level=debug msg="legolog: [INFO] [*.example.org] acme: Preparing to solve DNS-01"
traefik | time="2020-04-15T16:40:58Z" level=debug msg="legolog: [INFO] [*.example.org] acme: Trying to solve DNS-01"
traefik | time="2020-04-15T16:40:58Z" level=debug msg="legolog: [INFO] [*.example.org] acme: Checking DNS record propagation using [127.0.0.11:53]"
traefik | time="2020-04-15T16:40:58Z" level=debug msg="legolog: [INFO] Wait for propagation [timeout: 2m0s, interval: 2s]"
traefik | time="2020-04-15T16:40:58Z" level=debug msg="legolog: [INFO] [*.example.org] acme: Waiting for DNS record propagation."
traefik | time="2020-04-15T16:41:00Z" level=debug msg="legolog: [INFO] [*.example.org] acme: Waiting for DNS record propagation."
traefik | time="2020-04-15T16:41:02Z" level=debug msg="legolog: [INFO] [*.example.org] acme: Waiting for DNS record propagation."
traefik | time="2020-04-15T16:41:04Z" level=debug msg="legolog: [INFO] [*.example.org] acme: Waiting for DNS record propagation."
traefik | time="2020-04-15T16:41:06Z" level=debug msg="legolog: [INFO] [*.example.org] acme: Waiting for DNS record propagation."
traefik | time="2020-04-15T16:41:08Z" level=debug msg="legolog: [INFO] [*.example.org] acme: Waiting for DNS record propagation."
traefik | time="2020-04-15T16:41:10Z" level=debug msg="legolog: [INFO] [*.example.org] acme: Waiting for DNS record propagation."
traefik | time="2020-04-15T16:41:12Z" level=debug msg="legolog: [INFO] [*.example.org] acme: Waiting for DNS record propagation."
traefik | time="2020-04-15T16:41:14Z" level=debug msg="legolog: [INFO] [*.example.org] acme: Waiting for DNS record propagation."
traefik | time="2020-04-15T16:41:21Z" level=debug msg="legolog: [INFO] [*.example.org] acme: Cleaning DNS-01 challenge"
traefik | time="2020-04-15T16:41:22Z" level=debug msg="legolog: [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/id1"
traefik | time="2020-04-15T16:41:22Z" level=debug msg="legolog: [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/id1"
traefik | time="2020-04-15T16:41:22Z" level=debug msg="legolog: [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/id2"
traefik | time="2020-04-15T16:41:22Z" level=debug msg="legolog: [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/id2"
traefik | time="2020-04-15T16:41:22Z" level=error msg="Unable to obtain ACME certificate for domains \"example.org,*.example.org\" : unable to generate a certificate for the domains [example.org *.example.org]: acme: Error -> One or more domains had a problem:\n[*.example.org] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: During secondary validation: Incorrect TXT record \"null\" found at _acme-challenge.example.org, url: \n[example.org] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Connection refused, url: \n" providerName=wildcard-godaddy.acme
I do not understand either I'm misconfiguring something or if there's a problem on the let's encrypt/godaddy side.
Edit:
On port 80 I have another nginx instance up & running

Turned out it was a bug (fixed in v.2.2.1). See here: https://github.com/go-acme/lego/issues/1113

Related

LetsEncrypt lego script not working (Bitnami AWS Lightsail)

tl;dr: I used to use certbot to install a new certificate from LetsEncrypt, but that involved manually updating TXT records. So I tried to switch to lego to do it. But when I use lego to install a new certificate, it doesn't actually install a new certificate (old one still shows in browser).
Longer version:
I'm on an AWS Lightsail instance running Bitnami. I'm a little familiar with Linux, but I wouldn't consider myself "handy" with it.
In the past, I used to do the following to renew my certificates:
DOMAIN=mydomain.com
WILDCARD=*.$DOMAIN
sudo certbot -d $DOMAIN -d $WILDCARD --manual --preferred-challenges dns certonly
(Install TXT records)
sudo /opt/bitnami/ctlscript.sh restart apache
However, this is a bit annoying to have to do, and LetsEncrypt helpfully provides the lego library to do this automatically. So based off the documentation I could find, I did the following:
$ sudo /opt/bitnami/ctlscript.sh stop
Syntax OK
/opt/bitnami/apache2/scripts/ctl.sh : httpd stopped
/opt/bitnami/php/scripts/ctl.sh : php-fpm stopped
/opt/bitnami/mysql/scripts/ctl.sh : mysql stopped
$ sudo /opt/bitnami/letsencrypt/lego --tls --email="myemail#domain.com" --domains="mydomain.com" --domains="www.mydomain.com" --path="/opt/bitnami/letsencrypt" run
2020/02/28 16:58:57 [INFO] [mydomain.com, www.mydomain.com] acme: Obtaining bundled SAN certificate
2020/02/28 16:58:57 [INFO] [mydomain.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/3062019938
2020/02/28 16:58:57 [INFO] [www.mydomain.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/3062019945
2020/02/28 16:58:57 [INFO] [mydomain.com] acme: use tls-alpn-01 solver
2020/02/28 16:58:57 [INFO] [www.mydomain.com] acme: use tls-alpn-01 solver
2020/02/28 16:58:57 [INFO] [mydomain.com] acme: Trying to solve TLS-ALPN-01
2020/02/28 16:58:58 http: TLS handshake error from 74.108.143.17:49475: remote error: tls: illegal parameter
2020/02/28 16:58:58 http: TLS handshake error from 94.6.194.131:59130: remote error: tls: bad certificate
2020/02/28 16:58:58 http: TLS handshake error from 74.108.143.17:49476: remote error: tls: illegal parameter
2020/02/28 16:58:58 http: TLS handshake error from 94.6.194.131:59131: remote error: tls: bad certificate
2020/02/28 16:59:00 http: TLS handshake error from 81.187.9.124:49274: remote error: tls: bad certificate
2020/02/28 16:59:01 http: TLS handshake error from 74.108.143.17:49477: remote error: tls: illegal parameter
2020/02/28 16:59:01 http: TLS handshake error from 94.6.194.131:59136: remote error: tls: bad certificate
2020/02/28 16:59:03 [INFO] [mydomain.com] The server validated our request
2020/02/28 16:59:03 [INFO] [www.mydomain.com] acme: Trying to solve TLS-ALPN-01
2020/02/28 16:59:05 http: TLS handshake error from 94.6.194.131:59137: remote error: tls: bad certificate
2020/02/28 16:59:05 http: TLS handshake error from 74.108.143.17:49478: remote error: tls: illegal parameter
2020/02/28 16:59:06 http: TLS handshake error from 86.163.23.242:53113: remote error: tls: bad certificate
2020/02/28 16:59:08 http: TLS handshake error from 94.6.194.131:59138: remote error: tls: bad certificate
2020/02/28 16:59:08 http: TLS handshake error from 74.108.143.17:49479: remote error: tls: illegal parameter
2020/02/28 16:59:09 [INFO] [www.mydomain.com] The server validated our request
2020/02/28 16:59:09 [INFO] [mydomain.com, www.mydomain.com] acme: Validations succeeded; requesting certificates
2020/02/28 16:59:10 [INFO] [mydomain.com] Server responded with a certificate.
$ sudo /opt/bitnami/letsencrypt/lego --tls --email="myemail#domain.com" --domains="unmodchat.com" --domains="www.mydomain.com" --path="/opt/bitnami/letsencrypt" renew --days 90
2020/02/28 16:59:13 [INFO] [mydomain.com] acme: Trying renewal with 2158 hours remaining
2020/02/28 16:59:13 [INFO] [mydomain.com, www.mydomain.com] acme: Obtaining bundled SAN certificate
2020/02/28 16:59:14 [INFO] [mydomain.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/3062019938
2020/02/28 16:59:14 [INFO] [www.mydomain.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/3062019945
2020/02/28 16:59:14 [INFO] [mydomain.com] acme: authorization already valid; skipping challenge
2020/02/28 16:59:14 [INFO] [www.mydomain.com] acme: authorization already valid; skipping challenge
2020/02/28 16:59:14 [INFO] [mydomain.com, www.mydomain.com] acme: Validations succeeded; requesting certificates
2020/02/28 16:59:14 [INFO] [mydomain.com] Server responded with a certificate.
$ sudo /opt/bitnami/ctlscript.sh start
/opt/bitnami/mysql/scripts/ctl.sh : mysql started at port 3306
/opt/bitnami/php/scripts/ctl.sh : php-fpm started
Syntax OK
/opt/bitnami/apache2/scripts/ctl.sh : httpd started at port 80
After doing that, and restarting my browser, when I look at the certificate details it lists the same expiration date as before (March 20th, less than a month from now).
What am I doing wrong? I have a feeling that lego might be installing the certificates to the wrong place, but I'm not quite sure how to find where the "right" place is, nor how to tell lego to put them there.
Solution thanks to Jota Martos from Bitnami. The documentation I found wasn't complete, and skipped step 3 from this documentation: https://docs.bitnami.com/aws/how-to/generate-install-lets-encrypt-ssl/#step-3-configure-the-web-server-to-use-the-let-s-encrypt-certificate
After completing those mv commands, and using the lego command again, all is working as it should be now. Thanks Jota!

How to fix ... Unable to obtain ACME certificate ... thanks to rule \"Host:myhost.mydomain.com;PathPrefix:/app1{id:[0-9]?}\"

I have a set of meteor apps running as a docker stack along with traefik proxy, mongo and an http server. I had to do some redirection to pass traefik to each individual app so the client requests can be handled properly in response to the meteor ROOT URL. I do not understand the traefik log output that is telling me 'Unable to obtain ACME certificate for domains .... ' because of '... detected thanks to rule \"Host:myhost.mydomain.com;PathPrefix:/app2{id:[0-9]?}\"' Can someone please help me understand this log output? I am including the sanitized debug log, and sanitized traefik.toml and docker-compose.yml files. I don't think this is a bug, it is probably a misconfiguration.
I cannot use DNS challenge because I do not have control over the dns server. I have tried several configuration options. I suspect it has to do with the PathPrefix in the Host rule but don't think I understand enough about ACME to know how to properly change it.
Traefik.toml
logLevel = "DEBUG"
defaultEntryPoints = ["http", "https"]
[entryPoints]
[entryPoints.dashboard]
address = ":8090"
[entryPoints.dashboard.auth]
[entryPoints.dashboard.auth.basic]
users = ["admin:$2y$05$rd9MRJG/w0ugxIzmYy3L8.WpRheZfzPTTm17y.zq3cHKtZvMQ4OdW"]
[entryPoints.http]
address = ":80"
[entryPoints.http.redirect]
entryPoint = "https"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]
[entryPoints.https.redirect]
regex = "^(https://ip-205-156-8-94.ec2.internal)/?$"
replacement = "$1/"
permanent = true
[api]
entrypoint="dashboard"
[acme]
caServer = "https://acme-v02.api.letsencrypt.org/directory"
email = "myemail#mydomain.com"
storage = "acme.json"
OnHostRule = true
entryPoint = "https"
[acme.tlsChallenge]
[docker]
domain = "myhost.mydomain.com"
watch = true
network = "web"
exposedbydefualt = false
[traefikLog]
filePath = "/logs/traefik.log"
[accessLog]
filePath = "/logs/access.log"
***** docker-compose ****
version: "3.2"
networks:
web:
external: true
backend:
external: false
services:
traefik:
image: traefik
deploy:
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 3
window: 120s
networks:
- web
ports:
- "443:443"
- "80:80"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /home/myhome/container_deployment/traefik.toml:/traefik.toml
- /home/myhome/container_deployment/logs:/logs
- /home/myhome/container_deployment/acme.json:/acme.json
labels:
- traefik.frontend.rule=Host:myhost.mydomain.com;PathPrefixStrip:/proxy
- traefik.port=8090
mats-http:
image: myapps/production:mats-http
deploy:
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 3
window: 20s
volumes:
- /home/myhome/container_deployment/web:/web
labels:
- traefik.backend=mats-http/index.html
- traefik.frontend.rule=Host:myhost#mydomain.com;PathPrefixStrip:/
- traefik.docker.network=web
- traefik.port=8080
networks:
- web
mongo:
image: mongo
deploy:
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 3
window: 30s
command: -nojournal
ports:
- "27017:27017"
volumes:
- /home/myhome/mongodata:/data/db
networks:
- backend
- web
app1:
image: myapps/production:app1-2.2.0
deploy:
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 3
window: 60s
environment:
- DELAY=6
- ROOT_URL=https://myhost.mydomain.com/app1
volumes:
- /home/myhome/container_deployment/settings:/usr/app/settings
depends_on:
- mongo
labels:
- traefik.backend=app1
- traefik.frontend.rule=Host:myhost.mydomain.com;PathPrefix:/app1{id:[0-9]?}
- traefik.docker.network=web
- traefik.port=80
networks:
- web
- backend
app2:
image: myapps/production:app2-2.2.0
deploy:
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 3
window: 60s
environment:
- DELAY=6
- ROOT_URL=https://myhome.mydomain.com/app2
volumes:
- /home/myhome/container_deployment/settings:/usr/app/settings
depends_on:
- mongo
labels:
- traefik.backend=app2
- traefik.frontend.rule=Host:myhome.mydomain.com;PathPrefix:/app2{id:[0-9]?}
- traefik.docker.network=web
- traefik.port=80
networks:
- web
- backend
app3:
image: myapps/production:app3-2.2.0
deploy:
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 3
window: 60s
environment:
- DELAY=6
- ROOT_URL=https://myhome.mydomain.com/app3
volumes:
- /home/myhome/container_deployment/settings:/usr/app/settings
depends_on:
- mongo
labels:
- traefik.backend=app3
- traefik.frontend.rule=Host:myhome.mydomain.com;PathPrefix:/app3{id:[0-9]?}
- traefik.docker.network=web
- traefik.port=80
networks:
- web
- backend
***** truncated traefik debug log file *****
time="2019-07-11T16:03:38Z" level=info msg="Traefik version v1.7.12 built on 2019-05-29_07:35:02PM"
...
...
time="2019-07-11T16:03:38Z" level=debug msg="Configuration received from provider docker: {\"backends\":{\"backend-mats-http-index-html\":{\"servers\":{\"server-matsStack-mats-http-1-vpaeunxj6xif75dt61peb62an-695b347dcd588d1d0b320f01e5644738\":{\"url\":\"http://10.0.45.14:8080\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"backend-matsStack-mongo-1-kjoekf19fyw5ru0fr1azazzu5\":{\"servers\":{\"server-matsStack-mongo-1-kjoekf19fyw5ru0fr1azazzu5-e29723ab5c75dde0eaf988caf77e50b2\":{\"url\":\"http://10.0.45.3:27017\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"backend-matsStack-traefik-1-o1t2x6w1a0i3qu9nqwx6x67x1\":{\"servers\":{\"server-matsStack-traefik-1-o1t2x6w1a0i3qu9nqwx6x67x1-546c661a91789b6ce7fef697cc38e588\":{\"url\":\"http://10.0.45.12:8090\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"backend-app3\":{\"servers\":{\"server-matsStack-app3-1-71g3c7hr2qz1frc5paqn1y52i-382f1bea7ec466d09871b7dff5c5a47c\":{\"url\":\"http://10.0.45.8:80\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"backend-app2\":{\"servers\":{\"server-matsStack-app2-1-dvj9reft0nql50mp4jqxb9mx6-318c26e13ba26230fc29459a7f72c3aa\":{\"url\":\"http://10.0.45.10:80\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"backend-app1\":{\"servers\":{\"server-matsStack-app1-1-nk6ax8rfo9d3tly953huzrvb0-cb78098740da4a0710dfc1b9067e7842\":{\"url\":\"http://10.0.45.6:80\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}}},\"frontends\":{\"frontend-Host-matsStack-mongo-1-kjoekf19fyw5ru0fr1azazzu5-myhost.mydomain.com-4\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"backend-matsStack-mongo-1-kjoekf19fyw5ru0fr1azazzu5\",\"routes\":{\"route-frontend-Host-matsStack-mongo-1-kjoekf19fyw5ru0fr1azazzu5-myhost.mydomain.com-4\":{\"rule\":\"Host:matsStack-mongo.1.kjoekf19fyw5ru0fr1azazzu5.myhost.mydomain.com\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"frontend-Host-myhost.mydomain.com-PathPrefix-app3-id-0-9-3\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"backend-app3\",\"routes\":{\"route-frontend-Host-myhost.mydomain.com-PathPrefix-app3-id-0-9-3\":{\"rule\":\"Host:myhost.mydomain.com;PathPrefix:/app3{id:[0-9]?}\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"frontend-Host-myhost.mydomain.com-PathPrefix-app2-id-0-9-2\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"backend-app2\",\"routes\":{\"route-frontend-Host-myhost.mydomain.com-PathPrefix-app2-id-0-9-2\":{\"rule\":\"Host:myhost.mydomain.com;PathPrefix:/app2{id:[0-9]?}\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"frontend-Host-myhost.mydomain.com-PathPrefix-app1-id-0-9-5\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"backend-app1\",\"routes\":{\"route-frontend-Host-myhost.mydomain.com-PathPrefix-app1-id-0-9-5\":{\"rule\":\"Host:myhost.mydomain.com;PathPrefix:/app1{id:[0-9]?}\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"frontend-Host-myhost.mydomain.com-PathPrefixStrip-0\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"backend-mats-http-index-html\",\"routes\":{\"route-frontend-Host-myhost.mydomain.com-PathPrefixStrip-0\":{\"rule\":\"Host:myhost.mydomain.com;PathPrefixStrip:/\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"frontend-Host-myhost.mydomain.com-PathPrefixStrip-proxy-1\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"backend-matsStack-traefik-1-o1t2x6w1a0i3qu9nqwx6x67x1\",\"routes\":{\"route-frontend-Host-myhost.mydomain.com-PathPrefixStrip-proxy-1\":{\"rule\":\"Host:myhost.mydomain.com;PathPrefixStrip:/proxy\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null}}}"
time="2019-07-11T16:03:39Z" level=info msg="Server configuration reloaded on :80"
time="2019-07-11T16:03:39Z" level=info msg="Server configuration reloaded on :80"
...
...
time="2019-07-11T16:03:39Z" level=info msg="Server configuration reloaded on :443"
time="2019-07-11T16:03:39Z" level=info msg="Server configuration reloaded on :8090"
time="2019-07-11T16:03:39Z" level=debug msg="Try to challenge certificate for domain [myhost.mydomain.com] founded in Host rule"
time="2019-07-11T16:03:39Z" level=debug msg="Try to challenge certificate for domain [myhost.mydomain.com] founded in Host rule"
time="2019-07-11T16:03:39Z" level=debug msg="Try to challenge certificate for domain [myhost.mydomain.com] founded in Host rule"
time="2019-07-11T16:03:39Z" level=debug msg="Try to challenge certificate for domain [matsstack-mongo.1.kjoekf19fyw5ru0fr1azazzu5.myhost.mydomain.com] founded in Host rule"
time="2019-07-11T16:03:39Z" level=debug msg="Try to challenge certificate for domain [myhost.mydomain.com] founded in Host rule"
time="2019-07-11T16:03:39Z" level=debug msg="Try to challenge certificate for domain [myhost.mydomain.com] founded in Host rule"
time="2019-07-11T16:03:39Z" level=debug msg="Looking for provided certificate(s) to validate [\"myhost.mydomain.com\"]..."
time="2019-07-11T16:03:39Z" level=debug msg="Looking for provided certificate(s) to validate [\"myhost.mydomain.com\"]..."
time="2019-07-11T16:03:39Z" level=debug msg="Domains [\"myhost.mydomain.com\"] need ACME certificates generation for domains \"myhost.mydomain.com\"."
time="2019-07-11T16:03:39Z" level=debug msg="Domains [\"myhost.mydomain.com\"] need ACME certificates generation for domains \"myhost.mydomain.com\"."
time="2019-07-11T16:03:39Z" level=debug msg="Looking for provided certificate(s) to validate [\"myhost.mydomain.com\"]..."
time="2019-07-11T16:03:39Z" level=debug msg="Domains [\"myhost.mydomain.com\"] need ACME certificates generation for domains \"myhost.mydomain.com\"."
time="2019-07-11T16:03:39Z" level=debug msg="Looking for provided certificate(s) to validate [\"myhost.mydomain.com\"]..."
time="2019-07-11T16:03:39Z" level=debug msg="Domains [\"myhost.mydomain.com\"] need ACME certificates generation for domains \"myhost.mydomain.com\"."
time="2019-07-11T16:03:39Z" level=debug msg="Looking for provided certificate(s) to validate [\"matsstack-mongo.1.kjoekf19fyw5ru0fr1azazzu5.myhost.mydomain.com\"]..."
time="2019-07-11T16:03:39Z" level=debug msg="Domains [\"matsstack-mongo.1.kjoekf19fyw5ru0fr1azazzu5.myhost.mydomain.com\"] need ACME certificates generation for domains \"matsstack-mongo.1.kjoekf19fyw5ru0fr1azazzu5.myhost.mydomain.com\"."
time="2019-07-11T16:03:39Z" level=debug msg="Loading ACME certificates [myhost.mydomain.com]..."
time="2019-07-11T16:03:39Z" level=info msg="The key type is empty. Use default key type 4096."
time="2019-07-11T16:03:39Z" level=debug msg="Looking for provided certificate(s) to validate [\"myhost.mydomain.com\"]..."
time="2019-07-11T16:03:39Z" level=debug msg="No ACME certificate generation required for domains [\"myhost.mydomain.com\"]."
time="2019-07-11T16:03:39Z" level=debug msg="Loading ACME certificates [myhost.mydomain.com]..."
time="2019-07-11T16:03:39Z" level=debug msg="Loading ACME certificates [myhost.mydomain.com]..."
time="2019-07-11T16:03:39Z" level=debug msg="Loading ACME certificates [myhost.mydomain.com]..."
time="2019-07-11T16:03:39Z" level=debug msg="Loading ACME certificates [matsstack-mongo.1.kjoekf19fyw5ru0fr1azazzu5.myhost.mydomain.com]..."
time="2019-07-11T16:03:40Z" level=debug msg="Building ACME client..."
time="2019-07-11T16:03:40Z" level=debug msg="https://acme-v02.api.letsencrypt.org/directory"
time="2019-07-11T16:03:40Z" level=error msg="Unable to obtain ACME certificate for domains \"myhost.mydomain.com\" detected thanks to rule \"Host:myhost.mydomain.com;PathPrefixStrip:/\" : cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get https://acme-v02.api.letsencrypt.org/directory: dial tcp 23.55.128.36:443: connect: connection refused"
time="2019-07-11T16:03:40Z" level=debug msg="Building ACME client..."
time="2019-07-11T16:03:40Z" level=debug msg="https://acme-v02.api.letsencrypt.org/directory"
time="2019-07-11T16:03:40Z" level=error msg="Unable to obtain ACME certificate for domains \"myhost.mydomain.com\" detected thanks to rule \"Host:myhost.mydomain.com;PathPrefix:/app1{id:[0-9]?}\" : cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get https://acme-v02.api.letsencrypt.org/directory: dial tcp 23.55.128.36:443: connect: connection refused"
ime="2019-07-11T16:03:40Z" level=debug msg="Building ACME client..."
time="2019-07-11T16:03:40Z" level=debug msg="https://acme-v02.api.letsencrypt.org/directory"
time="2019-07-11T16:03:40Z" level=error msg="Unable to obtain ACME certificate for domains \"myhost.mydomain.com\" detected thanks to rule \"Host:myhost.mydomain.com;PathPrefix:/app3{id:[0-9]?}\" : cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get https://acme-v02.api.letsencrypt.org/directory: dial tcp 23.55.128.36:443: connect: connection refused"
time="2019-07-11T16:03:40Z" level=debug msg="Building ACME client..."
time="2019-07-11T16:03:40Z" level=debug msg="https://acme-v02.api.letsencrypt.org/directory"
time="2019-07-11T16:03:40Z" level=error msg="Unable to obtain ACME certificate for domains \"myhost.mydomain.com\" detected thanks to rule \"Host:myhost.mydomain.com;PathPrefix:/app2{id:[0-9]?}\" : cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get https://acme-v02.api.letsencrypt.org/directory: dial tcp 23.55.128.36:443: connect: connection refused"
time="2019-07-11T16:03:40Z" level=debug msg="Building ACME client..."
time="2019-07-11T16:03:40Z" level=debug msg="https://acme-v02.api.letsencrypt.org/directory"
time="2019-07-11T16:03:40Z" level=error msg="Unable to obtain ACME certificate for domains \"matsstack-mongo.1.kjoekf19fyw5ru0fr1azazzu5.myhost.mydomain.com\" detected thanks to rule \"Host:matsStack-mongo.1.kjoekf19fyw5ru0fr1azazzu5.myhost.mydomain.com\" : cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get https://acme-v02.api.letsencrypt.org/directory: dial tcp 23.55.128.36:443: connect: connection refused"
I expected the certificates to be obtained and for the challenge to work, instead SSL does not work properly.

traefik: hangs after msg="Server configuration reloaded on :80"

I am attempting to deploy a traefik app via docker-machine to an EC2 instance using the following commands:
docker-machine scp include/traefik.toml swarm-master:/home/ubuntu/traefik.toml
docker-machine scp include/acme.json swarm-master:/home/ubuntu/acme.json
docker $(docker-machine config swarm-master) run \
-v /var/run/docker.sock:/var/run/docker.sock \
-v $PWD/traefik.toml:/traefik.toml \
-v $PWD/acme.json:/acme.json \
-p 80:80 \
-p 443:443 \
-l traefik.frontend.rule=Host:traefik.domain.com \
-l traefik.port=8080 \
--network swarm-net \
--name traefik \
traefik:1.4.3-alpine \
-l DEBUG \
--docker
However, my application hangs after the following output:
traefik.toml 100% 503 29.4KB/s 00:00
acme.json 100% 0 0.0KB/s 00:00
time="2017-11-21T21:39:44Z" level=info msg="Traefik version v1.4.3 built on 2017-11-14_11:14:24AM"
time="2017-11-21T21:39:44Z" level=debug msg="Global configuration loaded {"GraceTimeOut":10000000000,"Debug":false,"CheckNewVersion":true,"AccessLogsFile":"","AccessLog":null,"TraefikLogsFile":"","LogLevel":"DEBUG","EntryPoints":{"http":{"Network":"","Address":":80","TLS":null,"Redirect":null,"Auth":null,"WhitelistSourceRange":null,"Compress":false,"ProxyProtocol":null,"ForwardedHeaders":{"Insecure":true,"TrustedIPs":null}}},"Cluster":null,"Constraints":[],"ACME":null,"DefaultEntryPoints":["http"],"ProvidersThrottleDuration":2000000000,"MaxIdleConnsPerHost":200,"IdleTimeout":0,"InsecureSkipVerify":false,"RootCAs":null,"Retry":null,"HealthCheck":{"Interval":30000000000},"RespondingTimeouts":null,"ForwardingTimeouts":null,"Docker":{"Watch":true,"Filename":"","Constraints":null,"Trace":false,"DebugLogGeneratedTemplate":false,"Endpoint":"unix:///var/run/docker.sock","Domain":"","TLS":null,"ExposedByDefault":true,"UseBindPortIP":false,"SwarmMode":false},"File":null,"Web":null,"Marathon":null,"Consul":null,"ConsulCatalog":null,"Etcd":null,"Zookeeper":null,"Boltdb":null,"Kubernetes":null,"Mesos":null,"Eureka":null,"ECS":null,"Rancher":null,"DynamoDB":null}"
time="2017-11-21T21:39:44Z" level=info msg="Preparing server http &{Network: Address::80 TLS:<nil> Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc42008bae0} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"
time="2017-11-21T21:39:44Z" level=info msg="Starting provider *docker.Provider {"Watch":true,"Filename":"","Constraints":null,"Trace":false,"DebugLogGeneratedTemplate":false,"Endpoint":"unix:///var/run/docker.sock","Domain":"","TLS":null,"ExposedByDefault":true,"UseBindPortIP":false,"SwarmMode":false}"
time="2017-11-21T21:39:44Z" level=info msg="Starting server on :80"
time="2017-11-21T21:39:44Z" level=debug msg="Provider connection established with docker 17.05.0-ce (API 1.29)"
time="2017-11-21T21:39:44Z" level=debug msg="Filtering container with empty frontend rule /swarm-agent"
time="2017-11-21T21:39:44Z" level=debug msg="Filtering container with empty frontend rule /swarm-agent-master"
time="2017-11-21T21:39:44Z" level=debug msg="Validation of load balancer method for backend backend-traefik failed: invalid load-balancing method ''. Using default method wrr."
time="2017-11-21T21:39:44Z" level=debug msg="Configuration received from provider docker: {"backends":{"backend-traefik":{"servers":{"server-traefik":{"url":"http://172.31.48.2:8080","weight":0}},"loadBalancer":{"method":"wrr"}}},"frontends":{"frontend-Host-traefik-domain-com-0":{"entryPoints":["http"],"backend":"backend-traefik","routes":{"route-frontend-Host-traefik-domain-com-0":{"rule":"Host:traefik.domain.com"}},"passHostHeader":true,"priority":0,"basicAuth":[],"headers":{}}}}"
time="2017-11-21T21:39:44Z" level=debug msg="Last docker config received more than 2s, OK"
time="2017-11-21T21:39:44Z" level=debug msg="Creating frontend frontend-Host-traefik-domain-com-0"
time="2017-11-21T21:39:44Z" level=debug msg="Wiring frontend frontend-Host-traefik-domain-com-0 to entryPoint http"
time="2017-11-21T21:39:44Z" level=debug msg="Creating route route-frontend-Host-traefik-domain-com-0 Host:traefik.domain.com"
time="2017-11-21T21:39:44Z" level=debug msg="Creating backend backend-traefik"
time="2017-11-21T21:39:44Z" level=debug msg="Creating load-balancer wrr"
time="2017-11-21T21:39:44Z" level=debug msg="Creating server server-traefik at http://172.31.48.2:8080 with weight 0"
time="2017-11-21T21:39:44Z" level=info msg="Server configuration reloaded on :80"
^Ctime="2017-11-21T21:39:59Z" level=info msg="I have to go... interrupt"
time="2017-11-21T21:39:59Z" level=info msg="Stopping server"
time="2017-11-21T21:39:59Z" level=debug msg="Waiting 10s seconds before killing connections on entrypoint http..."
time="2017-11-21T21:39:59Z" level=debug msg="Entrypoint http closed"
time="2017-11-21T21:39:59Z" level=info msg="Server stopped"
time="2017-11-21T21:39:59Z" level=info msg="Shutting down"
time="2017-11-21T21:39:59Z" level=error msg="Error creating server: http: Server closed"
traefik.toml:
debug = false
checkNewVersion = true
logLevel = "DEBUG"
defaultEntryPoints = ["http", "https"]
[web]
address = ":8080"
[web.auth.basic]
users = ["admin:alkdsjfalkdjflakdsjfalkdjfalkdjfaldkjf"]
[entryPoints]
[entryPoints.http]
address = ":80"
[entryPoints.http.redirect]
entryPoint = "https"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]
[acme]
email = "name#domain.com"
storage = "acme.json"
entryPoint = "https"
onHostRule = true
onDemand = false
and acme.json has permissions 600.
However, when I run the following (seemingly same) command after logging into my EC2 instance, traefik is successfully deployed:
docker run \
-v /var/run/docker.sock:/var/run/docker.sock \
-v $PWD/traefik.toml:/traefik.toml \
-v $PWD/acme.json:/acme.json \
-p 80:80 \
-p 443:443 \
-l traefik.frontend.rule=Host:traefik.domain.com \
-l traefik.port=8080 \
--network swarm-net \
--name traefik \
traefik:1.4.3-alpine \
-l DEBUG \
--docker
Any reason why I am not deploying with docker-machine are greatly appreciated. Thanks!

Why I can't forward port 8500 on host machine with consul image?

I try to run Consul image on Mac forwarding 8500 port for simple tests.
My command to run the image is:
docker run -it -p 8500:8500 consul agent -server -bootstrap 0.0.0.0
I do not use --net=host since it does not work on Mac so I try to forward 8500.
When I try to telnet from my Mac the connection gets immediately closed:
user$ telnet localhost 8500
Trying ::1...
Connected to localhost.
Escape character is '^]'.
Connection closed by foreign host.
Or when I try to add a new value I get:
consul kv put foo bar
Error! Failed writing data: Put http://127.0.0.1:8500/v1/kv/foo: dial tcp 127.0.0.1:8500: getsockopt: connection refused
What did I miss?
I have just tried what you have posted and seems that the port 8500 is opened
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f4ac8a5233e2 consul "docker-entrypoint..." 2 minutes ago Up 2 minutes 8300-8302/tcp, 8301-8302/udp, 8600/tcp, 8600/udp, 0.0.0.0:8500->8500/tcp sharp_knuth
And I get this:
Trying 0.0.0.0...
Connected to dev-consul
Escape character is '^]'.
Connection closed by foreign host.
However, it is running as you can see from the logs:
==> Starting Consul agent...
==> Consul agent running!
Version: 'v0.9.3'
Node ID: '27998add-58f9-e424-84a0-038db228629f'
Node name: '68bfdf141e7f'
Datacenter: 'dc1' (Segment: '<all>')
Server: true (Bootstrap: false)
Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2017/10/02 20:26:27 [DEBUG] Using random ID "27998add-58f9-e424-84a0-038db228629f" as node ID
2017/10/02 20:26:27 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:127.0.0.1:8300 Address:127.0.0.1:8300}]
2017/10/02 20:26:27 [INFO] raft: Node at 127.0.0.1:8300 [Follower] entering Follower state (Leader: "")
2017/10/02 20:26:27 [INFO] serf: EventMemberJoin: 68bfdf141e7f.dc1 127.0.0.1
2017/10/02 20:26:27 [INFO] serf: EventMemberJoin: 68bfdf141e7f 127.0.0.1
2017/10/02 20:26:27 [INFO] consul: Adding LAN server 68bfdf141e7f (Addr: tcp/127.0.0.1:8300) (DC: dc1)
2017/10/02 20:26:27 [INFO] consul: Handled member-join event for server "68bfdf141e7f.dc1" in area "wan"
2017/10/02 20:26:27 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2017/10/02 20:26:27 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
2017/10/02 20:26:27 [INFO] agent: Started HTTP server on [::]:8500
2017/10/02 20:26:27 [WARN] raft: Heartbeat timeout from "" reached, starting election
2017/10/02 20:26:27 [INFO] raft: Node at 127.0.0.1:8300 [Candidate] entering Candidate state in term 2
2017/10/02 20:26:27 [DEBUG] raft: Votes needed: 1
2017/10/02 20:26:27 [DEBUG] raft: Vote granted from 127.0.0.1:8300 in term 2. Tally: 1
2017/10/02 20:26:27 [INFO] raft: Election won. Tally: 1
2017/10/02 20:26:27 [INFO] raft: Node at 127.0.0.1:8300 [Leader] entering Leader state
2017/10/02 20:26:27 [INFO] consul: cluster leadership acquired
2017/10/02 20:26:27 [DEBUG] consul: Skipping self join check for "68bfdf141e7f" since the cluster is too small
2017/10/02 20:26:27 [INFO] consul: member '68bfdf141e7f' joined, marking health alive
2017/10/02 20:26:27 [INFO] consul: New leader elected: 68bfdf141e7f
2017/10/02 20:26:28 [INFO] agent: Synced node info
2017/10/02 20:27:27 [DEBUG] consul: Skipping self join check for "68bfdf141e7f" since the cluster is too small
2017/10/02 20:27:34 [DEBUG] agent: Node info in sync

EC2 Docker Consul clustering

I am sure that this answer is somewhere out there but I can not find or fix it after several tries. Here is the use-case :
1.> I have two ec2 instances belonging to the same VPC but having different security groups
2.> Both the security groups have 22,80 (for public) and All Traffic from all ports open for CIDR block 10.20.0.0/16
3.> The internal IP of the EC2 instances are 10.20.0.51 (server-1) and 10.20.0.202 (server-2)
4.> I am using these following commands to run two dockerized consul servers on them
server-1 : docker run -it -p 8400:8400 -p 8500:8500 -p 8600:53/udp -p 8301:8301 -p 8300:8300 -h node1 progrium/consul -server -advertise 10.20.0.51 -bootstrap-expect 2
server-2 : docker run -it -p 8400:8400 -p 8500:8500 -p 8600:53/udp -p 8301:8301 -p 8300:8300 --name node2 -h node2 progrium/consul -server -advertise 10.20.0.202 -join 10.20.0.51
5.> Both of them start and for one second they recognise each other and the election happens and the first node gets elected but soon after that server-2 starts saying "memberlist: Suspect node1 has failed, no acks received" and server-1 also says "memberlist: Suspect node2 has failed, no acks received"
This is what the logs look like for server-1
2016/01/04 19:18:35 [INFO] serf: EventMemberJoin: node2 10.20.0.202
2016/01/04 19:18:35 [INFO] consul: adding server node2 (Addr: 10.20.0.202:8300) (DC: dc1)
2016/01/04 19:18:35 [INFO] consul: Attempting bootstrap with nodes: [10.20.0.51:8300 10.20.0.202:8300]
2016/01/04 19:18:35 [WARN] raft: Heartbeat timeout reached, starting election
2016/01/04 19:18:35 [INFO] raft: Node at 10.20.0.51:8300 [Candidate] entering Candidate state
2016/01/04 19:18:35 [WARN] raft: Remote peer 10.20.0.202:8300 does not have local node 10.20.0.51:8300 as a peer
2016/01/04 19:18:35 [INFO] raft: Election won. Tally: 2
2016/01/04 19:18:35 [INFO] raft: Node at 10.20.0.51:8300 [Leader] entering Leader state
2016/01/04 19:18:35 [INFO] consul: cluster leadership acquired
2016/01/04 19:18:35 [INFO] consul: New leader elected: node1
2016/01/04 19:18:35 [INFO] raft: pipelining replication to peer 10.20.0.202:8300
2016/01/04 19:18:35 [INFO] consul: member 'node1' joined, marking health alive
2016/01/04 19:18:35 [INFO] consul: member 'node2' joined, marking health alive
2016/01/04 19:18:37 [INFO] memberlist: Suspect node2 has failed, no acks received
2016/01/04 19:18:37 [INFO] agent: Synced service 'consul'
2016/01/04 19:18:39 [INFO] memberlist: Suspect node2 has failed, no acks received
2016/01/04 19:18:41 [INFO] memberlist: Suspect node2 has failed, no acks received
2016/01/04 19:18:42 [INFO] memberlist: Marking node2 as failed, suspect timeout reached
2016/01/04 19:18:42 [INFO] serf: EventMemberFailed: node2 10.20.0.202
2016/01/04 19:18:42 [INFO] consul: removing server node2 (Addr: 10.20.0.202:8300) (DC: dc1)
And for server -2
2016/01/04 19:18:10 [INFO] serf: EventMemberJoin: node2 10.20.0.202
2016/01/04 19:18:10 [INFO] serf: EventMemberJoin: node2.dc1 10.20.0.202
2016/01/04 19:18:10 [INFO] raft: Node at 10.20.0.202:8300 [Follower] entering Follower state
2016/01/04 19:18:10 [INFO] agent: (LAN) joining: [10.20.0.51]
2016/01/04 19:18:10 [INFO] consul: adding server node2 (Addr: 10.20.0.202:8300) (DC: dc1)
2016/01/04 19:18:10 [INFO] consul: adding server node2.dc1 (Addr: 10.20.0.202:8300) (DC: dc1)
2016/01/04 19:18:10 [INFO] serf: EventMemberJoin: node1 10.20.0.51
2016/01/04 19:18:10 [INFO] agent: (LAN) joined: 1 Err: <nil>
2016/01/04 19:18:10 [ERR] agent: failed to sync remote state: No cluster leader
2016/01/04 19:18:10 [INFO] consul: adding server node1 (Addr: 10.20.0.51:8300) (DC: dc1)
2016/01/04 19:18:12 [INFO] memberlist: Suspect node1 has failed, no acks received
2016/01/04 19:18:14 [INFO] memberlist: Suspect node1 has failed, no acks received
2016/01/04 19:18:16 [INFO] memberlist: Suspect node1 has failed, no acks received
2016/01/04 19:18:17 [INFO] memberlist: Marking node1 as failed, suspect timeout reached
2016/01/04 19:18:17 [INFO] serf: EventMemberFailed: node1 10.20.0.51
2016/01/04 19:18:17 [INFO] memberlist: Suspect node1 has failed, no acks received
2016/01/04 19:18:17 [INFO] consul: removing server node1 (Addr: 10.20.0.51:8300) (DC: dc1)
2016/01/04 19:18:19 [INFO] serf: EventMemberJoin: node1 10.20.0.51
2016/01/04 19:18:19 [INFO] consul: adding server node1 (Addr: 10.20.0.51:8300) (DC: dc1)
2016/01/04 19:18:19 [INFO] consul: New leader elected: node1
2016/01/04 19:18:21 [INFO] memberlist: Suspect node1 has failed, no acks received
2016/01/04 19:18:22 [INFO] agent: Synced service 'consul'
2016/01/04 19:18:23 [INFO] memberlist: Suspect node1 has failed, no acks received
2016/01/04 19:18:25 [INFO] memberlist: Suspect node1 has failed, no acks received
2016/01/04 19:18:26 [INFO] memberlist: Marking node1 as failed, suspect timeout reached
2016/01/04 19:18:26 [INFO] serf: EventMemberFailed: node1 10.20.0.51
2016/01/04 19:18:26 [INFO] consul: removing server node1 (Addr: 10.20.0.51:8300) (DC: dc1)
2016/01/04 19:18:26 [INFO] memberlist: Suspect node1 has failed, no acks received
2016/01/04 19:18:40 [INFO] serf: attempting reconnect to node1 10.20.0.51:8301
2016/01/04 19:18:40 [INFO] serf: EventMemberJoin: node1 10.20.0.51
What exactly I am doing wrong. All I want is to run two consul docker in two EC2 instances and communicate between them without explicitly opening up the ports in the security group (When I explicitly open them up it works of course!)
Please can somebody help.
Thanks

Resources