Read docker env variable inside Spring application.yaml - spring-boot

I'm setting an environment variable inside my docker-compose.yaml file and want to use that variable's value inside my Spring Boot's application.yaml. I was told that doing something like
app:
auth:
tokenSecret: tokensecretvaluehere
tokenExpirationMsec: 864000000
oauth2:
sso:
url: ${SSO_URL}
(where SSO_URL is defined in my docker-compose.yaml) in my Spring application.yaml. However, this causes an error when I run docker-compose up --build because it can't find that variable (error is like: Could not resolve placeholder SSO_URL in value "${SSO_URL}"). This is an example of what my docker-compose.yaml:
api:
restart: always
ports:
- "8080:8080"
links:
- redis
- db
environment:
- SERVER_SERVLET_SESSION_COOKIE_DOMAIN=localhost
- SSO_URL=myvaluehere
I was asked to not uses the System.getenv functions in Java and instead set the variable like above. From there I would just use the #Value annotation to get it in my Java code as like below:
#Value("${app.oauth2.sso.url}")
private String ssoUrl;
This is more of the application.yaml:
heb:
togglr:
jwt:
secret:
id: 101
session:
seconds: 600
tokenheader: X-TOGGLR-TOKEN
logging:
level:
com:
heb: debug
default: debug
path: logs
server:
error:
whitelabel:
enabled: false
port: 8080
servlet:
context-path: /togglr-api
use-forward-headers: true
spring:
application:
name: togglr_api
freemarker:
enabled: false
groovy:
template:
enabled: false
jmx:
enabled: false
main:
banner-mode: 'off'
thymeleaf:
cache: false
security:
oauth2:
client:
registration:
github:
clientId:
clientSecret:
redirectUri:
scope:
- user:email
- read:user
app:
auth:
tokenSecret:
tokenExpirationMsec: 864000000
oauth2:
sso:
url: ${SSO_URL}

In general spring boot applications are able to read the Environment variables accessible in the docker container. Its stated in the documentation (see item 10 in the list at the very beginning of the document).
So the problem might be elsewhere:
It might be a typo in the question, but if you're using application.yaml as opposed to application properties, then you should have something like:
sso:
url: ${SSO_URL}
Make sure, that the env variable SSO_URL is indeed accessible in the container even before you start the spring boot application. In java (for debugging purposes only) you can do something like:
#SpringBootApplication
public class MyApp {
public static void main(String [] args) {
System.out.println(System.getenv("SSO_URL"));
SpringApplication.run(MyApp.class);
}
}

Solution was to not use an underscore character in the variable name.

I feel what you are missing is the build context within the docker-compose file. Or, you have multiple profile based application.yml and not the correct profile is being set.
Below is the working code / config. So we are creating a springboot application from docker-compose, where docker-compose builds the image of springboot application and passes the required environment variables.
Snippet from Spring application.yml where we are using the env variable named API_BASE and TEST_API_PATH
third-party-api:
base-url: ${API_BASE}
test-api-path: ${TEST_API_PATH}
Below is the snippet from docker-compose.yml
my-app:
image: my-app-image-name
build:
dockerfile: Dockerfile
context: .
ports:
- '9080:8080'
environment:
API_BASE: http://mock-api:8080
TEST_API_PATH: /someTestApiPath
SPRING_PROFILES_ACTIVE: dev
Docker file of my application is pretty simple.
FROM openjdk:17-alpine
ARG JAR_FILE=build/libs/*.jar
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java","-jar","/app.jar"]
Dockerfile and docker-compose.yml are on the same hierarcy, if there is any difference in your structure then it should reflect in below config of docker-compose.yml
build:
dockerfile: Dockerfile

Related

spring boot: how to verify env variables?

Is it possible to check env vars on startup in Spring Boot?
For example config looks like this:
spring:
data:
mongodb:
uri: ${MONGODB_URI}
rabbitmq:
addresses: ${RMQ_CLUSTER_ADDRESSES}
I want the app to not start if one of var is not exists.

Spring application properties (map) from docker environment

I would like to configure my dockerized spring boot application using Docker Environments. The property is a map, which I configure it in application.yml like below.
spring:
kafka:
producer:
properties:
"schema.registry.url": http://schema-registry.com:8081
I tried the following, but it didn't worked,
environment:
- SPRING_KAFKA_PRODUCER_PROPERTIES_SCHEMA.REGISTRY.URL=http://schema-registry.com:8081
How can I configure this schema.registry.url parameter from docker environment ?
Well, first of all, I would require a little bit more of information about how do you use that containerized application: Do you deploy it with docker-compose? Does it forms part of a Docker Swarm?
Depending of this, the possible solutions can vary.
Docker Swarm
For example, if you are using Docker Swarm, you can define your application.yml as a template:
application.yml.template
spring:
kafka:
producer:
properties:
"schema.registry.url": {{ env "schema_registry" }}
Then, you will have to parse that template. For that, I will suppose you have located your Spring Boot executable JAR under /usr/app in the container and that your image is named springboot-app.
docker-compose.yml
version: "3.8"
services:
springboot-app:
image: springboot-app:latest
environment:
SPRING_KAFKA_PRODUCER_PROPERTIES_SCHEMA.REGISTRY.URL: 'http://schema-registry.com:8081'
configs:
- source: springboot-app.application.yml
target: /usr/app/config/application.yaml
mode: 0440
configs:
springboot-app.application.yml:
template_driver: golang
file: ./application.yml.template
So you can deploy now your Swarm with docker stack deploy -c docker-compose.yml springboot-app.
Or even better, if you are working in a production environment, you can separate the environment variables from the common configuration:
docker-compose.yml
version: "3.8"
services:
springboot-app:
image: springboot-app:latest
configs:
- source: springboot-app.application.yml
target: /usr/app/config/application.yaml
mode: 0440
configs:
springboot-app.application.yml:
template_driver: golang
file: ./application.yml.template
docker-compose.dev.yml
version: "3.8"
services:
springboot-app:
environment:
SPRING_KAFKA_PRODUCER_PROPERTIES_SCHEMA.REGISTRY.URL: 'http://schema-registry.com:8081'
And deploy it as docker stack deploy -c docker-compose.yml -c docker-compose.dev.yml springboot-app.
Docker Compose
Since you mentioned in a lately comment that you are using docker-compose, the way of working with isn't the same.
First of all, not all the properties in Spring can be overridden in the Docker Compose file, only the ones that you can pass to Maven at the time of building or starting the application.
Also, it seems you have wrongly defined the environment Property, since normally all those that you provide you should change the dots '.' by underscores '_', but anyway, since normally the configuration of a Kafka Producer goes further than just defining an URL, I would use the profiles feature of Spring.
You can create several profiles with the configuration combinations that you want, and inform Spring via Compose which one you want to use. Let's see an example.
application.yml
spring:
config:
activate:
on-profile: "development"
kafka:
producer:
properties:
"schema.registry.url": https://kafka-dev-endpoint.com
---
spring:
config:
activate:
on-profile: "production"
kafka:
producer:
properties:
"schema.registry.url": https://kafka-prod-endpoint.com
and finally then:
docker-compose.yml
environment:
- SPRING_PROFILES_ACTIVE=development
If you wanna check further, you have more information about that here: https://docs.spring.io/spring-boot/docs/current/reference/html/howto.html#howto-set-active-spring-profiles

Passing list of YAML properties from Docker compose to Spring Boot application

My spring Boot application contains the following list of properties in application.yml:
hs:
targets:
- name: system_a
url: https://system-a
username: username-a
password: password-a
- name: system_b
url: https://system-b
username: username-b
password: password-b
Instead of defining them there, I'd like to pass them from Docker Compose via docker-compose.yml.
my-app:
image: my-image
ports:
- 9002:9002
environment:
SPRING_PROFILES_ACTIVE: dev
HS_TARGETS_NAME: system-a
HS_TARGETS_URL: https://system-a
HS_TARGETS_USERNAME: username_a
HS_TARGETS_PASSWORD: password_a
HS_TARGETS_NAME: system-b
HS_TARGETS_URL: https://system-b
HS_TARGETS_USERNAME: username_b
HS_TARGETS_PASSWORD: password_b
But I doubt that this will work because the variables all have the same name. Is there any way to do that or do I have to restructore my Spring Boot properties?
Pass the properties as
HS_TARGETS_0_NAME: system-a
HS_TARGETS_0_URL: https://system-a
HS_TARGETS_0_USERNAME: username_a
HS_TARGETS_0_PASSWORD: password_a
HS_TARGETS_1_NAME: system-b
HS_TARGETS_1_URL: https://system-b
HS_TARGETS_1_USERNAME: username_b
HS_TARGETS_1_PASSWORD: password_b
Tested with Spring Boot 2.1.17

Spring Config server not reachable with docker-compose until client is restarted

I have a 'discovery first' setup with Eureka, Config Server, and my client.
The issue is that these 3 services start in order, but the client-server seems to register too early, and can never find config-server. I've tried a third-party library that allows a wait until config-server:8888 is available, but that doesn't always seem to work either. It's similar to a race condition.
The workaround is that if I docker restart the client-server after everything is up, it registers and finds config-server just fine.
First run of docker-compose:
Fetching config from server at : http://localhost:8888
Connect Timeout Exception on Url - http://localhost:8888. Will be trying the next url if available
When I docker restart the client:
Fetching config from server at : http://a80b001d04a7:8888/
Located environment: name=client-server, profiles=[default], label=null, version=053c8e1b14dc0281d5af0349c9b2cf012c1a346f, state=null
Not sure if my JAVA_OPTS properties aren't being set fast enough from my docker-compose.yml, or there is some networking race condition, or what. I've been going back and forth on this for too long.
My configuration is below:
Here's my docker-compose.yml:
version: '3'
services:
eureka:
image: eureka-server:latest
environment:
- "JAVA_OPTS=-DEUREKA_SERVER=http://eureka:8761/eureka"
ports:
- 8761:8761
config:
image: config-server:latest
environment:
- "JAVA_OPTS=-DEUREKA_SERVER=http://eureka:8761/eureka"
depends_on:
- eureka
ports:
- 8888:8888
client:
image: client-server:latest
environment:
JAVA_OPTS: -DEUREKA_SERVER=http://eureka:8761/eureka
depends_on:
- config
ports:
- 9000:9000
Here's the eureka-server application.yml:
server:
port: 8761
spring:
application:
name: eureka-server
eureka:
client:
registerWithEureka: false
fetchRegistry: false
service-url:
defaultZone: ${EUREKA_SERVER:http://localhost:8761/eureka}
Here's the config-server bootstrap.yml:
server:
port: 8888
eureka:
client:
serviceUrl:
defaultZone: ${EUREKA_SERVER:http://localhost:8761/eureka}
spring:
application:
name: config-server
Here's the client-server bootstrap.yml:
spring:
application:
name: client-server
cloud:
config:
discovery:
enabled: true
serviceId: config-server
fast-fail: true
retry:
max-attempts: 10000
max-interval: 1000
eureka:
instance:
hostname: client-server
client:
registerWithEureka: true
fetchRegistry: true
serviceUrl:
defaultZone: ${EUREKA_SERVER:http://localhost:8761/eureka}
Edit:
Using the docker-compose wait library (https://github.com/ufoscout/docker-compose-wait), I can have the client-server wait for eureka and config to be available, then wait 90 seconds (Eureka documentation suggests that registration could take up to 90 seconds), and it seems to work consistently.
Is this an acceptable solution? Feels like a bit of a hack.
Being purist the answer to your question is NO, it is not an acceptable solution, because as it is stated here, Docker removed healthcheck from v3 on for some reason:
Docker have made a conscious decision not to support features that wait for containers to be in a "ready" state. They argue that applications depending on other systems should be resilient to failure.
In the same link, it is described why:
The problem of waiting for a database (for example) to be ready is really just a subset of a much larger problem of distributed systems. In production, your database could become unavailable or move hosts at any time. Your application needs to be resilient to these types of failures.
To handle this, your application should attempt to re-establish a connection to the database after a failure. If the application retries the connection, it should eventually be able to connect to the database.
Basically then, there are three options:
Use v2.1 with healhcheck. See an example here
Use v3 and a tool like wait-for-it or
dockerize as #ortomala-lokni already perfectly explained
Make your application resilient to config-server failure and able config-client to retry the connection on startup
The recommended and acceptable solution is 3). You can use Spring Retry as it is mentioned here. Find below the bootstrap.yml configuration:
spring:
application:
name: config-client
profiles:
active: dev
cloud:
config:
discovery:
enabled: true
service-id: config-server
fail-fast: true
retry:
initial-interval: 1500
multiplier: 1.5
max-attempts: 10000
max-interval: 1000
eureka:
instance:
hostname: config-client
client:
registerWithEureka: true
fetchRegistry: true
serviceUrl:
defaultZone: ${EUREKA_SERVER:http://localhost:8761/eureka}
BTW I found an error in your spring configuration. It is fail-fast and not fast-fail.
Remember to include the following dependencies (or similar if you are using gradle):
<dependency>
<groupId>org.springframework.retry</groupId>
<artifactId>spring-retry</artifactId>
</dependency>
You can find a very well configuration (and explanation) here taking also into account resiliency during the registering process in the Eureka Server.
When having a microservices environment we must think of the resiliency of our environment when platform services like config-service, discovery-service are not available for a short period of time.
But I am not a purist at all and I would not have removed some functionality people is using (it is a question of freedom). So, an alternative solution is:
If it is working for you, then go ahead
Because I do not really understand why Docker suppressed the fantastic healthcheck command from v3.
Service dependency are always tricky when using docker-compose.
Your solution is acceptable because "there is no other way".
To avoid third-part libs, this is what I do in the same scenario:
In the Dockerfile I add netcat-openbsd, a bash file I call entrypoint and the application jar and then I run the entrypoint.sh.
FROM openjdk:8-jdk-alpine
RUN apk --no-cache add netcat-openbsd
COPY entrypoint.sh /opt/bin/
COPY app.jar /opt/lib/
RUN chmod 755 /opt/esusab-bi/bin/app/entrypoint.sh
The entrypoint file has the following instruction:
#!/bin/sh
while ! nc -z config 8888 ; do
echo "Waiting for upcoming Config Server"
sleep 2
done
java -jar /opt/lib/app.jar
It will delay the application start-up until your config server is up, without a specific interval.
The best solution is probably, as Carlos Cavero said, to make your application resilient to config-server failure. But you can also solve the problem by using the wait-for script from Eficode on Github.
Copy the script into your container and in your docker-compose.yml use:
client:
image: client-server:latest
environment:
JAVA_OPTS: -DEUREKA_SERVER=http://eureka:8761/eureka
depends_on:
- config
ports:
- 9000:9000
command: wait-for $CONFIGSERVER_SERVICE_NAME:$CONFIGSERVER_PORT -- java $JVM_OPTIONS -jar client.war $SPRING_OPTIONS
The environment variables for CONFIGSERVER_SERVICE_NAME and CONFIGSERVER_PORT can be defined in your Docker Compose environment file.
If you need to wait for multiple services, you can merge this pull request and list all needed services in the command line parameters such as:
command: wait-for $SERVICE1_NAME $SERVICE1_PORT $SERVICE2_NAME $SERVICE2_PORT -- java $JVM_OPTIONS -jar client.war $SPRING_OPTIONS
Just a friendly tip: You should not bind Config to Eureka but the other way around -> Eureka should be Config client.

how to make spring boot client's application.yml get value from config server

Is there anyway to make spring cloud config client's application.yml read values from spring config server?
For example,
on my spring cloud config client, the application.yml is like this
spring:
application:
name: clienttest
mvc:
view:
prefix: /jsp/
suffix: .jsp
server:
port: 8080
context-path: /clienttest
tomcat:
uri-encoding: UTF-8
eureka:
client:
service-url: {"defaultZone":"http://dev.euraka01.app.com:8769/eureka/,http://dev.euraka02.app.com:8770/eureka/"}
instance:
prefer-ip-address: true
and my bootstrap.yml file is as below
spring:
application:
name: clienttest
cloud:
config:
name: clienttest
uri: http://192.168.2.101:9000
enabled: true
profile: out_test
label: master
now for the service-url value, for different environment, I have to config different eureka url values, my question is that, is there anyway that I can config the service-url value in the config server? like I set the value as ${service-url} in the application.yml, and when I start the config client server, it get the value from the config server according the profile and label which I set in the bootstrap.yml.
You can look up properties on the config server by both profile and label, where label is either either a branch, tag.
/{application}/{profile}[/{label}]
/{application}-{profile}.yml
/{label}/{application}-{profile}.yml
/{application}-{profile}.properties
/{label}/{application}-{profile}.properties
In your example above your config server will try and find a file named
clienttest-out_test.properties
In the git repo on the master branch.
spring:
application:
name: clienttest
cloud:
config:
profile: out_test
label: master
See the example and also a good doc here
Essex Boy,
Thank you very much for your help, and I was able to read the value from different config profile before.
My question is how to make application.yml get the value from config server, and now I've solve it by myself, the answer is quite easy, in the application, set value like ${service-url}, the full answer is as below:
in my application.yml, the content is as below:
spring:
application:
name: clienttest
server:
port: 8080
context-path: /clienttest
tomcat:
uri-encoding: UTF-8
eureka:
client:
service-url: {"defaultZone":"${service-url}"}
instance:
prefer-ip-address: true
Please note the service-url value, now the value is set as {"defaultZone":"${service-url}"}, and in my application.properties file which on the config server, the properties file content is as below:
service-url=http://192.168.2.101:8769/eureka/,http://192.168.2.101:8770/eureka/
then when I start the mocroservice, it could resist it self on the http://192.168.2.101:8769/eureka/ and http://192.168.2.101:8770/eureka/
which is what result I want.

Resources