Spring Config server not reachable with docker-compose until client is restarted - spring

Question

I have a 'discovery first' setup with Eureka, Config Server, and my client.
The issue is that these 3 services start in order, but the client-server seems to register too early, and can never find config-server. I've tried a third-party library that allows a wait until config-server:8888 is available, but that doesn't always seem to work either. It's similar to a race condition.
The workaround is that if I docker restart the client-server after everything is up, it registers and finds config-server just fine.
First run of docker-compose:
Fetching config from server at : http://localhost:8888
Connect Timeout Exception on Url - http://localhost:8888. Will be trying the next url if available
When I docker restart the client:
Fetching config from server at : http://a80b001d04a7:8888/
Located environment: name=client-server, profiles=[default], label=null, version=053c8e1b14dc0281d5af0349c9b2cf012c1a346f, state=null
Not sure if my JAVA_OPTS properties aren't being set fast enough from my docker-compose.yml, or there is some networking race condition, or what. I've been going back and forth on this for too long.
My configuration is below:
Here's my docker-compose.yml:
version: '3'
services:
eureka:
image: eureka-server:latest
environment:
- "JAVA_OPTS=-DEUREKA_SERVER=http://eureka:8761/eureka"
ports:
- 8761:8761
config:
image: config-server:latest
environment:
- "JAVA_OPTS=-DEUREKA_SERVER=http://eureka:8761/eureka"
depends_on:
- eureka
ports:
- 8888:8888
client:
image: client-server:latest
environment:
JAVA_OPTS: -DEUREKA_SERVER=http://eureka:8761/eureka
depends_on:
- config
ports:
- 9000:9000
Here's the eureka-server application.yml:
server:
port: 8761
spring:
application:
name: eureka-server
eureka:
client:
registerWithEureka: false
fetchRegistry: false
service-url:
defaultZone: ${EUREKA_SERVER:http://localhost:8761/eureka}
Here's the config-server bootstrap.yml:
server:
port: 8888
eureka:
client:
serviceUrl:
defaultZone: ${EUREKA_SERVER:http://localhost:8761/eureka}
spring:
application:
name: config-server
Here's the client-server bootstrap.yml:
spring:
application:
name: client-server
cloud:
config:
discovery:
enabled: true
serviceId: config-server
fast-fail: true
retry:
max-attempts: 10000
max-interval: 1000
eureka:
instance:
hostname: client-server
client:
registerWithEureka: true
fetchRegistry: true
serviceUrl:
defaultZone: ${EUREKA_SERVER:http://localhost:8761/eureka}
Edit:
Using the docker-compose wait library (https://github.com/ufoscout/docker-compose-wait), I can have the client-server wait for eureka and config to be available, then wait 90 seconds (Eureka documentation suggests that registration could take up to 90 seconds), and it seems to work consistently.
Is this an acceptable solution? Feels like a bit of a hack.

Answer 1

Being purist the answer to your question is NO, it is not an acceptable solution, because as it is stated here, Docker removed healthcheck from v3 on for some reason:
Docker have made a conscious decision not to support features that wait for containers to be in a "ready" state. They argue that applications depending on other systems should be resilient to failure.
In the same link, it is described why:
The problem of waiting for a database (for example) to be ready is really just a subset of a much larger problem of distributed systems. In production, your database could become unavailable or move hosts at any time. Your application needs to be resilient to these types of failures.
To handle this, your application should attempt to re-establish a connection to the database after a failure. If the application retries the connection, it should eventually be able to connect to the database.
Basically then, there are three options:
Use v2.1 with healhcheck. See an example here
Use v3 and a tool like wait-for-it or
dockerize as #ortomala-lokni already perfectly explained
Make your application resilient to config-server failure and able config-client to retry the connection on startup
The recommended and acceptable solution is 3). You can use Spring Retry as it is mentioned here. Find below the bootstrap.yml configuration:
spring:
application:
name: config-client
profiles:
active: dev
cloud:
config:
discovery:
enabled: true
service-id: config-server
fail-fast: true
retry:
initial-interval: 1500
multiplier: 1.5
max-attempts: 10000
max-interval: 1000
eureka:
instance:
hostname: config-client
client:
registerWithEureka: true
fetchRegistry: true
serviceUrl:
defaultZone: ${EUREKA_SERVER:http://localhost:8761/eureka}
BTW I found an error in your spring configuration. It is fail-fast and not fast-fail.
Remember to include the following dependencies (or similar if you are using gradle):
<dependency>
<groupId>org.springframework.retry</groupId>
<artifactId>spring-retry</artifactId>
</dependency>
You can find a very well configuration (and explanation) here taking also into account resiliency during the registering process in the Eureka Server.
When having a microservices environment we must think of the resiliency of our environment when platform services like config-service, discovery-service are not available for a short period of time.
But I am not a purist at all and I would not have removed some functionality people is using (it is a question of freedom). So, an alternative solution is:
If it is working for you, then go ahead
Because I do not really understand why Docker suppressed the fantastic healthcheck command from v3.

Answer 2

Service dependency are always tricky when using docker-compose.
Your solution is acceptable because "there is no other way".
To avoid third-part libs, this is what I do in the same scenario:
In the Dockerfile I add netcat-openbsd, a bash file I call entrypoint and the application jar and then I run the entrypoint.sh.
FROM openjdk:8-jdk-alpine
RUN apk --no-cache add netcat-openbsd
COPY entrypoint.sh /opt/bin/
COPY app.jar /opt/lib/
RUN chmod 755 /opt/esusab-bi/bin/app/entrypoint.sh
The entrypoint file has the following instruction:
#!/bin/sh
while ! nc -z config 8888 ; do
echo "Waiting for upcoming Config Server"
sleep 2
done
java -jar /opt/lib/app.jar
It will delay the application start-up until your config server is up, without a specific interval.

Answer 3

The best solution is probably, as Carlos Cavero said, to make your application resilient to config-server failure. But you can also solve the problem by using the wait-for script from Eficode on Github.
Copy the script into your container and in your docker-compose.yml use:
client:
image: client-server:latest
environment:
JAVA_OPTS: -DEUREKA_SERVER=http://eureka:8761/eureka
depends_on:
- config
ports:
- 9000:9000
command: wait-for $CONFIGSERVER_SERVICE_NAME:$CONFIGSERVER_PORT -- java $JVM_OPTIONS -jar client.war $SPRING_OPTIONS
The environment variables for CONFIGSERVER_SERVICE_NAME and CONFIGSERVER_PORT can be defined in your Docker Compose environment file.
If you need to wait for multiple services, you can merge this pull request and list all needed services in the command line parameters such as:
command: wait-for $SERVICE1_NAME $SERVICE1_PORT $SERVICE2_NAME $SERVICE2_PORT -- java $JVM_OPTIONS -jar client.war $SPRING_OPTIONS

Answer 4

Just a friendly tip: You should not bind Config to Eureka but the other way around -> Eureka should be Config client.