How to fix ... Unable to obtain ACME certificate ... thanks to rule \"Host:myhost.mydomain.com;PathPrefix:/app1{id:[0-9]?}\" - lets-encrypt

Question

I have a set of meteor apps running as a docker stack along with traefik proxy, mongo and an http server. I had to do some redirection to pass traefik to each individual app so the client requests can be handled properly in response to the meteor ROOT URL. I do not understand the traefik log output that is telling me 'Unable to obtain ACME certificate for domains .... ' because of '... detected thanks to rule \"Host:myhost.mydomain.com;PathPrefix:/app2{id:[0-9]?}\"' Can someone please help me understand this log output? I am including the sanitized debug log, and sanitized traefik.toml and docker-compose.yml files. I don't think this is a bug, it is probably a misconfiguration.
I cannot use DNS challenge because I do not have control over the dns server. I have tried several configuration options. I suspect it has to do with the PathPrefix in the Host rule but don't think I understand enough about ACME to know how to properly change it.
Traefik.toml
logLevel = "DEBUG"
defaultEntryPoints = ["http", "https"]
[entryPoints]
[entryPoints.dashboard]
address = ":8090"
[entryPoints.dashboard.auth]
[entryPoints.dashboard.auth.basic]
users = ["admin:$2y$05$rd9MRJG/w0ugxIzmYy3L8.WpRheZfzPTTm17y.zq3cHKtZvMQ4OdW"]
[entryPoints.http]
address = ":80"
[entryPoints.http.redirect]
entryPoint = "https"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]
[entryPoints.https.redirect]
regex = "^(https://ip-205-156-8-94.ec2.internal)/?$"
replacement = "$1/"
permanent = true
[api]
entrypoint="dashboard"
[acme]
caServer = "https://acme-v02.api.letsencrypt.org/directory"
email = "myemail#mydomain.com"
storage = "acme.json"
OnHostRule = true
entryPoint = "https"
[acme.tlsChallenge]
[docker]
domain = "myhost.mydomain.com"
watch = true
network = "web"
exposedbydefualt = false
[traefikLog]
filePath = "/logs/traefik.log"
[accessLog]
filePath = "/logs/access.log"
***** docker-compose ****
version: "3.2"
networks:
web:
external: true
backend:
external: false
services:
traefik:
image: traefik
deploy:
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 3
window: 120s
networks:
- web
ports:
- "443:443"
- "80:80"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /home/myhome/container_deployment/traefik.toml:/traefik.toml
- /home/myhome/container_deployment/logs:/logs
- /home/myhome/container_deployment/acme.json:/acme.json
labels:
- traefik.frontend.rule=Host:myhost.mydomain.com;PathPrefixStrip:/proxy
- traefik.port=8090
mats-http:
image: myapps/production:mats-http
deploy:
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 3
window: 20s
volumes:
- /home/myhome/container_deployment/web:/web
labels:
- traefik.backend=mats-http/index.html
- traefik.frontend.rule=Host:myhost#mydomain.com;PathPrefixStrip:/
- traefik.docker.network=web
- traefik.port=8080
networks:
- web
mongo:
image: mongo
deploy:
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 3
window: 30s
command: -nojournal
ports:
- "27017:27017"
volumes:
- /home/myhome/mongodata:/data/db
networks:
- backend
- web
app1:
image: myapps/production:app1-2.2.0
deploy:
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 3
window: 60s
environment:
- DELAY=6
- ROOT_URL=https://myhost.mydomain.com/app1
volumes:
- /home/myhome/container_deployment/settings:/usr/app/settings
depends_on:
- mongo
labels:
- traefik.backend=app1
- traefik.frontend.rule=Host:myhost.mydomain.com;PathPrefix:/app1{id:[0-9]?}
- traefik.docker.network=web
- traefik.port=80
networks:
- web
- backend
app2:
image: myapps/production:app2-2.2.0
deploy:
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 3
window: 60s
environment:
- DELAY=6
- ROOT_URL=https://myhome.mydomain.com/app2
volumes:
- /home/myhome/container_deployment/settings:/usr/app/settings
depends_on:
- mongo
labels:
- traefik.backend=app2
- traefik.frontend.rule=Host:myhome.mydomain.com;PathPrefix:/app2{id:[0-9]?}
- traefik.docker.network=web
- traefik.port=80
networks:
- web
- backend
app3:
image: myapps/production:app3-2.2.0
deploy:
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 3
window: 60s
environment:
- DELAY=6
- ROOT_URL=https://myhome.mydomain.com/app3
volumes:
- /home/myhome/container_deployment/settings:/usr/app/settings
depends_on:
- mongo
labels:
- traefik.backend=app3
- traefik.frontend.rule=Host:myhome.mydomain.com;PathPrefix:/app3{id:[0-9]?}
- traefik.docker.network=web
- traefik.port=80
networks:
- web
- backend
***** truncated traefik debug log file *****
time="2019-07-11T16:03:38Z" level=info msg="Traefik version v1.7.12 built on 2019-05-29_07:35:02PM"
...
...
time="2019-07-11T16:03:38Z" level=debug msg="Configuration received from provider docker: {\"backends\":{\"backend-mats-http-index-html\":{\"servers\":{\"server-matsStack-mats-http-1-vpaeunxj6xif75dt61peb62an-695b347dcd588d1d0b320f01e5644738\":{\"url\":\"http://10.0.45.14:8080\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"backend-matsStack-mongo-1-kjoekf19fyw5ru0fr1azazzu5\":{\"servers\":{\"server-matsStack-mongo-1-kjoekf19fyw5ru0fr1azazzu5-e29723ab5c75dde0eaf988caf77e50b2\":{\"url\":\"http://10.0.45.3:27017\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"backend-matsStack-traefik-1-o1t2x6w1a0i3qu9nqwx6x67x1\":{\"servers\":{\"server-matsStack-traefik-1-o1t2x6w1a0i3qu9nqwx6x67x1-546c661a91789b6ce7fef697cc38e588\":{\"url\":\"http://10.0.45.12:8090\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"backend-app3\":{\"servers\":{\"server-matsStack-app3-1-71g3c7hr2qz1frc5paqn1y52i-382f1bea7ec466d09871b7dff5c5a47c\":{\"url\":\"http://10.0.45.8:80\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"backend-app2\":{\"servers\":{\"server-matsStack-app2-1-dvj9reft0nql50mp4jqxb9mx6-318c26e13ba26230fc29459a7f72c3aa\":{\"url\":\"http://10.0.45.10:80\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"backend-app1\":{\"servers\":{\"server-matsStack-app1-1-nk6ax8rfo9d3tly953huzrvb0-cb78098740da4a0710dfc1b9067e7842\":{\"url\":\"http://10.0.45.6:80\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}}},\"frontends\":{\"frontend-Host-matsStack-mongo-1-kjoekf19fyw5ru0fr1azazzu5-myhost.mydomain.com-4\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"backend-matsStack-mongo-1-kjoekf19fyw5ru0fr1azazzu5\",\"routes\":{\"route-frontend-Host-matsStack-mongo-1-kjoekf19fyw5ru0fr1azazzu5-myhost.mydomain.com-4\":{\"rule\":\"Host:matsStack-mongo.1.kjoekf19fyw5ru0fr1azazzu5.myhost.mydomain.com\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"frontend-Host-myhost.mydomain.com-PathPrefix-app3-id-0-9-3\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"backend-app3\",\"routes\":{\"route-frontend-Host-myhost.mydomain.com-PathPrefix-app3-id-0-9-3\":{\"rule\":\"Host:myhost.mydomain.com;PathPrefix:/app3{id:[0-9]?}\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"frontend-Host-myhost.mydomain.com-PathPrefix-app2-id-0-9-2\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"backend-app2\",\"routes\":{\"route-frontend-Host-myhost.mydomain.com-PathPrefix-app2-id-0-9-2\":{\"rule\":\"Host:myhost.mydomain.com;PathPrefix:/app2{id:[0-9]?}\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"frontend-Host-myhost.mydomain.com-PathPrefix-app1-id-0-9-5\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"backend-app1\",\"routes\":{\"route-frontend-Host-myhost.mydomain.com-PathPrefix-app1-id-0-9-5\":{\"rule\":\"Host:myhost.mydomain.com;PathPrefix:/app1{id:[0-9]?}\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"frontend-Host-myhost.mydomain.com-PathPrefixStrip-0\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"backend-mats-http-index-html\",\"routes\":{\"route-frontend-Host-myhost.mydomain.com-PathPrefixStrip-0\":{\"rule\":\"Host:myhost.mydomain.com;PathPrefixStrip:/\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"frontend-Host-myhost.mydomain.com-PathPrefixStrip-proxy-1\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"backend-matsStack-traefik-1-o1t2x6w1a0i3qu9nqwx6x67x1\",\"routes\":{\"route-frontend-Host-myhost.mydomain.com-PathPrefixStrip-proxy-1\":{\"rule\":\"Host:myhost.mydomain.com;PathPrefixStrip:/proxy\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null}}}"
time="2019-07-11T16:03:39Z" level=info msg="Server configuration reloaded on :80"
time="2019-07-11T16:03:39Z" level=info msg="Server configuration reloaded on :80"
...
...
time="2019-07-11T16:03:39Z" level=info msg="Server configuration reloaded on :443"
time="2019-07-11T16:03:39Z" level=info msg="Server configuration reloaded on :8090"
time="2019-07-11T16:03:39Z" level=debug msg="Try to challenge certificate for domain [myhost.mydomain.com] founded in Host rule"
time="2019-07-11T16:03:39Z" level=debug msg="Try to challenge certificate for domain [myhost.mydomain.com] founded in Host rule"
time="2019-07-11T16:03:39Z" level=debug msg="Try to challenge certificate for domain [myhost.mydomain.com] founded in Host rule"
time="2019-07-11T16:03:39Z" level=debug msg="Try to challenge certificate for domain [matsstack-mongo.1.kjoekf19fyw5ru0fr1azazzu5.myhost.mydomain.com] founded in Host rule"
time="2019-07-11T16:03:39Z" level=debug msg="Try to challenge certificate for domain [myhost.mydomain.com] founded in Host rule"
time="2019-07-11T16:03:39Z" level=debug msg="Try to challenge certificate for domain [myhost.mydomain.com] founded in Host rule"
time="2019-07-11T16:03:39Z" level=debug msg="Looking for provided certificate(s) to validate [\"myhost.mydomain.com\"]..."
time="2019-07-11T16:03:39Z" level=debug msg="Looking for provided certificate(s) to validate [\"myhost.mydomain.com\"]..."
time="2019-07-11T16:03:39Z" level=debug msg="Domains [\"myhost.mydomain.com\"] need ACME certificates generation for domains \"myhost.mydomain.com\"."
time="2019-07-11T16:03:39Z" level=debug msg="Domains [\"myhost.mydomain.com\"] need ACME certificates generation for domains \"myhost.mydomain.com\"."
time="2019-07-11T16:03:39Z" level=debug msg="Looking for provided certificate(s) to validate [\"myhost.mydomain.com\"]..."
time="2019-07-11T16:03:39Z" level=debug msg="Domains [\"myhost.mydomain.com\"] need ACME certificates generation for domains \"myhost.mydomain.com\"."
time="2019-07-11T16:03:39Z" level=debug msg="Looking for provided certificate(s) to validate [\"myhost.mydomain.com\"]..."
time="2019-07-11T16:03:39Z" level=debug msg="Domains [\"myhost.mydomain.com\"] need ACME certificates generation for domains \"myhost.mydomain.com\"."
time="2019-07-11T16:03:39Z" level=debug msg="Looking for provided certificate(s) to validate [\"matsstack-mongo.1.kjoekf19fyw5ru0fr1azazzu5.myhost.mydomain.com\"]..."
time="2019-07-11T16:03:39Z" level=debug msg="Domains [\"matsstack-mongo.1.kjoekf19fyw5ru0fr1azazzu5.myhost.mydomain.com\"] need ACME certificates generation for domains \"matsstack-mongo.1.kjoekf19fyw5ru0fr1azazzu5.myhost.mydomain.com\"."
time="2019-07-11T16:03:39Z" level=debug msg="Loading ACME certificates [myhost.mydomain.com]..."
time="2019-07-11T16:03:39Z" level=info msg="The key type is empty. Use default key type 4096."
time="2019-07-11T16:03:39Z" level=debug msg="Looking for provided certificate(s) to validate [\"myhost.mydomain.com\"]..."
time="2019-07-11T16:03:39Z" level=debug msg="No ACME certificate generation required for domains [\"myhost.mydomain.com\"]."
time="2019-07-11T16:03:39Z" level=debug msg="Loading ACME certificates [myhost.mydomain.com]..."
time="2019-07-11T16:03:39Z" level=debug msg="Loading ACME certificates [myhost.mydomain.com]..."
time="2019-07-11T16:03:39Z" level=debug msg="Loading ACME certificates [myhost.mydomain.com]..."
time="2019-07-11T16:03:39Z" level=debug msg="Loading ACME certificates [matsstack-mongo.1.kjoekf19fyw5ru0fr1azazzu5.myhost.mydomain.com]..."
time="2019-07-11T16:03:40Z" level=debug msg="Building ACME client..."
time="2019-07-11T16:03:40Z" level=debug msg="https://acme-v02.api.letsencrypt.org/directory"
time="2019-07-11T16:03:40Z" level=error msg="Unable to obtain ACME certificate for domains \"myhost.mydomain.com\" detected thanks to rule \"Host:myhost.mydomain.com;PathPrefixStrip:/\" : cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get https://acme-v02.api.letsencrypt.org/directory: dial tcp 23.55.128.36:443: connect: connection refused"
time="2019-07-11T16:03:40Z" level=debug msg="Building ACME client..."
time="2019-07-11T16:03:40Z" level=debug msg="https://acme-v02.api.letsencrypt.org/directory"
time="2019-07-11T16:03:40Z" level=error msg="Unable to obtain ACME certificate for domains \"myhost.mydomain.com\" detected thanks to rule \"Host:myhost.mydomain.com;PathPrefix:/app1{id:[0-9]?}\" : cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get https://acme-v02.api.letsencrypt.org/directory: dial tcp 23.55.128.36:443: connect: connection refused"
ime="2019-07-11T16:03:40Z" level=debug msg="Building ACME client..."
time="2019-07-11T16:03:40Z" level=debug msg="https://acme-v02.api.letsencrypt.org/directory"
time="2019-07-11T16:03:40Z" level=error msg="Unable to obtain ACME certificate for domains \"myhost.mydomain.com\" detected thanks to rule \"Host:myhost.mydomain.com;PathPrefix:/app3{id:[0-9]?}\" : cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get https://acme-v02.api.letsencrypt.org/directory: dial tcp 23.55.128.36:443: connect: connection refused"
time="2019-07-11T16:03:40Z" level=debug msg="Building ACME client..."
time="2019-07-11T16:03:40Z" level=debug msg="https://acme-v02.api.letsencrypt.org/directory"
time="2019-07-11T16:03:40Z" level=error msg="Unable to obtain ACME certificate for domains \"myhost.mydomain.com\" detected thanks to rule \"Host:myhost.mydomain.com;PathPrefix:/app2{id:[0-9]?}\" : cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get https://acme-v02.api.letsencrypt.org/directory: dial tcp 23.55.128.36:443: connect: connection refused"
time="2019-07-11T16:03:40Z" level=debug msg="Building ACME client..."
time="2019-07-11T16:03:40Z" level=debug msg="https://acme-v02.api.letsencrypt.org/directory"
time="2019-07-11T16:03:40Z" level=error msg="Unable to obtain ACME certificate for domains \"matsstack-mongo.1.kjoekf19fyw5ru0fr1azazzu5.myhost.mydomain.com\" detected thanks to rule \"Host:matsStack-mongo.1.kjoekf19fyw5ru0fr1azazzu5.myhost.mydomain.com\" : cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get https://acme-v02.api.letsencrypt.org/directory: dial tcp 23.55.128.36:443: connect: connection refused"
I expected the certificates to be obtained and for the challenge to work, instead SSL does not work properly.