Spring Boot Admin with Docker Swarm Unable to pull Health Metrics - spring-boot

Question

I am trying to get spring boot admin to work in a docker swarm cluster using a zookeeper discovery service mechanism to ensure that all clients are dynamically discovered once connected to zookeeper. Problem is it appears springboot admin is unable to reach the health actuator endpoints on the clients due to a connection refused even though all docker services are using the same overlay network and each container can ping one another which ive verified via docker exec -it ping to ensure they are all reachable from one another.
Ive also verified that the clients and the admin service's are properly connecting to zookeeper and that zookeeper + admin dashboard are infact seeing those clients as registered.
To recreate this issue ive created a simple docker compose that deploys two spring boot admin apps with the actuators enabled over the same overlay network via the compose file below:
version: '3.1'
services:
zoo1:
image: zookeeper:3.4.12
hostname: zoo1
networks:
- nsp_test
deploy:
restart_policy:
condition: on-failure
placement:
constraints: [node.hostname == nj51nreda5v]
environment:
ZOO_MY_ID: 1
ZOO_SERVERS: server.1=0.0.0.0:2888:3888 server.2=zoo2:2888:3888
zoo2:
image: zookeeper:3.4.12
hostname: zoo2
networks:
- nsp_test
deploy:
restart_policy:
condition: on-failure
placement:
constraints: [node.hostname == nj51nreda6v]
environment:
ZOO_MY_ID: 2
ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=0.0.0.0:2888:3888
nspadmin:
image: admin:77
ports:
- "9084:8080"
networks:
- nsp_test
depends_on:
- "zoo1"
- "zoo2"
deploy:
restart_policy:
condition: on-failure
mode: global
environment:
ZK_HOST: zoo1:2181,zoo2:2182
SPRING_PROFILES_ACTIVE: ssldev
networks:
nsp_test:
external:
name: nsp_test
From this configuration I see both spring admin dashboards registered in zookeeper and display as OFFLINE (since it cant reach the /health actuator)
The following two addresses are what it registers for the clients in SBA.
https://10.255.0.19:8080/ OFFLINE
https://10.255.0.20:8080/ OFFLINE
The exception I get.
2018-12-31 04:20:31.926 INFO 1 --- [ updateTask1] d.c.boot.admin.registry.StatusUpdater : Couldn't retrieve status for Application [id=28eab1e1, name=nsp-admin, managementUrl=https://10.255.0.20:8080/, healthUrl=https://10.255.0.20:8080/health, serviceUrl=https://10.255.0.20:8080/]
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://10.255.0.20:8080/health": Connect to 10.255.0.20:8080 [/10.255.0.20] failed: connect timed out; nested exception is org.apache.http.conn.ConnectTimeoutException: Connect to 10.255.0.20:8080 [/10.255.0.20] failed: connect timed out
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:666) ~[spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:628) ~[spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:549) ~[spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
at de.codecentric.boot.admin.web.client.ApplicationOperations.doGet(ApplicationOperations.java:68) ~[spring-boot-admin-server-1.5.6.jar!/:1.5.6]
at de.codecentric.boot.admin.web.client.ApplicationOperations.getHealth(ApplicationOperations.java:58) ~[spring-boot-admin-server-1.5.6.jar!/:1.5.6]
at de.codecentric.boot.admin.registry.StatusUpdater.queryStatus(StatusUpdater.java:111) [spring-boot-admin-server-1.5.6.jar!/:1.5.6]
at de.codecentric.boot.admin.registry.StatusUpdater.updateStatus(StatusUpdater.java:65) [spring-boot-admin-server-1.5.6.jar!/:1.5.6]
at de.codecentric.boot.admin.registry.StatusUpdateApplicationListener$1.run(StatusUpdateApplicationListener.java:47) [spring-boot-admin-server-1.5.6.jar!/:1.5.6]
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_151]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_151]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_151]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to 10.255.0.20:8080 [/10.255.0.20] failed: connect timed out
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) ~[httpclient-4.5.3.jar!/:4.5.3]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) ~[httpclient-4.5.3.jar!/:4.5.3]
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[httpclient-4.5.3.jar!/:4.5.3]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[httpclient-4.5.3.jar!/:4.5.3]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[httpclient-4.5.3.jar!/:4.5.3]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.3.jar!/:4.5.3]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[httpclient-4.5.3.jar!/:4.5.3]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.3.jar!/:4.5.3]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.3.jar!/:4.5.3]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.3.jar!/:4.5.3]
at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:89) ~[spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53) ~[spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:652) ~[spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
... 15 common frames omitted
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_151]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_151]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_151]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_151]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_151]
at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_151]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) ~[httpclient-4.5.3.jar!/:4.5.3]
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.3.jar!/:4.5.3]
My SBA Configuration yml
server:
port: 8080
spring:
boot:
admin:
client:
prefer-ip: false
datasource:
driverClassName: org.postgresql.Driver
url: ${DB_URL}
username: ${DB_USER}
password: ${DB_PASSWORD}
application:
name: nsp-admin
cloud:
config:
discovery:
enabled: true
zookeeper:
connect-string: ${ZK_HOST}
discovery:
uri-spec: https://{address}:{port}
metadata:
management:
context-path: /
health:
path: /health
management:
security:
enabled: false
security:
basic:
enabled: false
#security.require-ssl: true
server.ssl.enabled: true
server.ssl.key-store-type: PKCS12
server.ssl.key-store: *****
server.ssl.key-store-password: *****
UPDATE
After debugging the issue a bit more I realized it without a doubt has to do with the hostname/IP the clients are registering in zookeeper.
When i perform a curl using the docker id's as the hostnames the /health api returns when performing the curl from SBA to the Client container id.
This Works:
docker exec -it 8403c5001b9e curl -k https://bf41c73af594:8080/health
This does not work results in a timeout: docker exec -it 8403c5001b9e curl -k https://10.255.0.20:8080/health
Is it possible to force zookeeper to register the hostname or the containerid instead?
UPDATE
Setting spring.cloud.zookeeper.discovery.instanceHost: ${HOSTNAME} in my application.yml resolves the issue. It forces the correct containerId to be registered to zookeeper.

Answer 1

You don't need to do all these circuses. In Docker, there is a concept called service discovery. It is a local DNS resolution taken care by docker.
You can either use the container name or specify an alias instead of IP/container id as these will change every time.
Method 1:
By default, docker adds network name with service name to name a container. You can fix a name to the container by using container_name keyword in decker-compose. Then you can use that name instead of IP. This will get the respective containers resolved.
An example compose file:
version: '3.1'
services:
zoo1:
image: zookeeper:3.4.12
hostname: zoo1
container_name: zoo1
networks:
- nsp_test
deploy:
restart_policy:
condition: on-failure
placement:
constraints: [node.hostname == nj51nreda5v]
environment:
ZOO_MY_ID: 1
ZOO_SERVERS: server.1=0.0.0.0:2888:3888 server.2=zoo2:2888:3888
zoo2:
image: zookeeper:3.4.12
hostname: zoo2
container_name: zoo2
networks:
- nsp_test
deploy:
restart_policy:
condition: on-failure
placement:
constraints: [node.hostname == nj51nreda6v]
environment:
ZOO_MY_ID: 2
ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=0.0.0.0:2888:3888
nspadmin:
image: admin:77
ports:
- "9084:8080"
networks:
- nsp_test
depends_on:
- "zoo1"
- "zoo2"
deploy:
restart_policy:
condition: on-failure
mode: global
environment:
ZK_HOST: zoo1:2181,zoo2:2182
SPRING_PROFILES_ACTIVE: ssldev
networks:
nsp_test:
external:
name: nsp_test
Now you can reach zoo1 and zoo2 as zoo1,zoo2. Not suitable for swarm mode as container_name is ignored
Method 2: (Recommended for docker swarm mode)
You can specify alias for each host and can access that service by using the alias.
An example compose file:
version: '3.1'
services:
zoo1:
image: zookeeper:3.4.12
hostname: zoo1
networks:
default:
aliases:
- zoo1
- zoo.1
deploy:
restart_policy:
condition: on-failure
placement:
constraints: [node.hostname == nj51nreda5v]
environment:
ZOO_MY_ID: 1
ZOO_SERVERS: server.1=0.0.0.0:2888:3888 server.2=zoo2:2888:3888
zoo2:
image: zookeeper:3.4.12
hostname: zoo2
networks:
default:
aliases:
- zoo2
- zoo.2
deploy:
restart_policy:
condition: on-failure
placement:
constraints: [node.hostname == nj51nreda6v]
environment:
ZOO_MY_ID: 2
ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=0.0.0.0:2888:3888
nspadmin:
image: admin:77
ports:
- "9084:8080"
networks:
- nsp_test
depends_on:
- "zoo1"
- "zoo2"
deploy:
restart_policy:
condition: on-failure
mode: global
environment:
ZK_HOST: zoo1:2181,zoo2:2182
SPRING_PROFILES_ACTIVE: ssldev
networks:
default:
external:
name: nsp_test
Here zoo1 can be resolved as zoo1,zoo.1,zoo1.nsp_test,zoo.1.nsp_test. The same goes for zoo2. suitable for swarm mode as well.
Method 3:
If you know what is the name of the service that is getting created then you can use that as well to resolve the container.
For example:
version: '3.1'
services:
zoo1:
image: zookeeper:3.4.12
hostname: zoo1
networks:
- nsp_test
deploy:
restart_policy:
condition: on-failure
placement:
constraints: [node.hostname == nj51nreda5v]
environment:
ZOO_MY_ID: 1
ZOO_SERVERS: server.1=0.0.0.0:2888:3888 server.2=zoo2:2888:3888
zoo2:
image: zookeeper:3.4.12
hostname: zoo2
networks:
- nsp_test
deploy:
restart_policy:
condition: on-failure
placement:
constraints: [node.hostname == nj51nreda6v]
environment:
ZOO_MY_ID: 2
ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=0.0.0.0:2888:3888
nspadmin:
image: admin:77
ports:
- "9084:8080"
networks:
- nsp_test
depends_on:
- "zoo1"
- "zoo2"
deploy:
restart_policy:
condition: on-failure
mode: global
environment:
ZK_HOST: zoo1:2181,zoo2:2182
SPRING_PROFILES_ACTIVE: ssldev
networks:
nsp_test:
external:
name: nsp_test
Let us assume that the above config creates containers with name zoo1_nsp_test and zoo2_nsp_test. You can resolve the containers by using these name as well. Not suitable for swarm node as container name differs from host to host.
Note:
All the above methods work only if the containers are connected to same network.
References:
Compose file version 3 reference#container_name
Compose file version 3 reference#aliases
service discovery
Load balancing, service discovery and security