Spring boot 2.4.x cannot handle multi document yml files from config server - spring-boot

Java version: 8
Spring Boot version: 2.4.1
Spring Cloud version: 2020.0.0, specifically I use a Spring Cloud Config Server connected to GIT and our services are Spring Cloud Config Clients.
I have migrated away from using bootstrap.yml and started using spring.config.import and spring.config.activate.on-profile as mentioned in the documentation here and here
My configuration in my service, who is a client to the config server looks like this:
server.port: 9001
spring:
application.name: my-rest-service
config.import: configserver:http://localhost:8888
cloud.config.profile: ${spring.profiles.active}
My configuration in the config server looks like this:
application.yml (has two documents separated by the ---)
logging:
file.name: <omitted>
level:
root: INFO
---
spring:
config.activate.on-profile: dev
logging.level.root: DEBUG
my-rest-sercive.yml (has two documents separated by the ---)
spring:
datasource:
driver-class-name: <omitted>
username: <omitted>
password: <omitted>
---
spring:
config.activate.on-profile: dev
datasource.url: <omitted>
Because there is a profile "dev" active, I successfully get the following 4 configurations from config server:
application.yml: general logging level
application.yml: specific logging for dev
my-rest-sercive.yml: general datasource properties
my-rest-sercive.yml: specific datasource url for dev
I can see these 4 sources successfully being fetched when I use my browser or when I debug or in the logs when I lower the loglevel to trace:
o.s.b.c.config.ConfigDataEnvironment : Adding imported property source 'configserver:https://git.company.com/path.git/file:C:\configservergit\config\my-rest-service.yml'
o.s.b.c.config.ConfigDataEnvironment : Adding imported property source 'configserver:https://git.company.com/path.git/file:C:\configservergit\config\my-rest-service.yml'
o.s.b.c.config.ConfigDataEnvironment : Adding imported property source 'configserver:https://git.company.com/path.git/file:C:\configservergit\config\application.yml'
o.s.b.c.config.ConfigDataEnvironment : Adding imported property source 'configserver:https://git.company.com/path.git/file:C:\configservergit\config\application.yml'
However, notice that because I use multi document yml files, out of these 4 property sources only TWO unique names are used.
In a later step, when Spring creates the data source bean, he complains he cannot find the data source URL. If I debug the spring bean factory I can indeed see that out of the 4 property files returned by the config server, only two have remained (the ones that don't contain the dev profile specific configuration). I assume this is because they have an identical name and they overwrite each other. This is an effect of this piece of code in the MutablePropertySource.class:
public void addLast(PropertySource<?> propertySource) {
synchronized(this.propertySourceList) {
this.removeIfPresent(propertySource); <-- this is the culrprit!
this.propertySourceList.add(propertySource);
}
}
This is a breaking change from Spring 2.3/Spring Cloud Hoxton where it correctly collected all properties. I think spring cloud needs to change the config server so that every document within a yml has has a unique name when returned to Spring. This is exactly how Spring Boot handles multi document yml files, by appending the String (documenyt #1) to the property source name
I found an interesting note about profiles and multi document yml, basically saying it is not supported, but this doesn't apply to my use case because my yml files are not profiles based (there is no -{profileName} in the last part of the file name).

This is a known issue with the new release. We can track the issue here on the spring cloud config server github page.
The workaround seems to be stop using multi document yml files and use multiple distinct files with the profile name in the filename.

Related

In which class in the source code of spring-boot or spring is the application.yml or application.properties file processed?

In which class in the source code of spring-boot or spring is the application.yml file or application.properties processed?
For spring boot (version 2.x) the application properties are loaded from the environment into the context via a PropertySourceLoader.
In for example the spring-boot-2.6.3.jar we can find the following file:
META-INF/spring.factories
# PropertySource Loaders
org.springframework.boot.env.PropertySourceLoader=\
org.springframework.boot.env.PropertiesPropertySourceLoader,\
org.springframework.boot.env.YamlPropertySourceLoader
Where PropertiesPropertySourceLoader loads .properties and .xml files, and YamlPropertySourceLoader loads .yml and .yaml.
These are loaded with the SpringFactoriesLoader, which we can see in action in org.springframework.boot.context.config.ConfigFileApplicationListener (deprecated) or org.springframework.boot.context.config.StandardConfigDataLocationResolver (via ConfigDataEnvironmentPostProcessor -> ConfigDataEnvironment -> ConfigDataLocationResolvers) :
this.propertySourceLoaders = SpringFactoriesLoader.loadFactories(PropertySourceLoader.class,
getClass().getClassLoader());
You can read in the ConfigFileApplicationListener JavaDoc that the properties are indeed loaded with this class:
EnvironmentPostProcessor that configures the context environment by loading properties from well known file locations. By default properties will be loaded from 'application.properties' and/or 'application.yml' files in the following locations:
file:./config/
file:./config/*/
file:./
classpath:config/
classpath:
...
If you're interested in context loading from the environment in spring(boot), I suggest you setup your project with maven, download the sources jars, and have a look around in the mentioned factories file. You will find more relevant code in the org.springframework.boot.env and org.springframework.boot.context (config and properties) packages.
You can find your application.yml or application.properties at the src/main/resources. You can have as many as possible configurations for your spring boot application for every case. Lets assume that you have 3 local-profiles like demo, production and server, so you made 3 configuration and assumingyou set for active profile the demo at the application.yml . I hope you get the idea. Its the first thing that actually is running before the springboot is up.
Please look the officials docs !

Different YAML configuration file for junit test using an Externalized configuration in Spring Boot

I am following a tutorial on using external configuration files for Spring Boot. I got everything to work exactly as intended but I'm having issues overriding the default YAML config for my tests.
Could someone please point me in the right direction or advice if using '#PropertySource' is the best way to load config files into the project (There is a bunch of properties and I would like to keep the application.yaml as clean as possible)
Project Structure:
src: - main/resources/foo.yml <-- always loads this one
- test/resources/foo.yml <-- never loads
What I tried:
#PropertySource(value = "classpath:foo.yml")
Doesn't load test/resoruces/foo.yml to the classpath
ActiveProfiles()
How I usually change config properties but in this case, it's not a profile so it doesn't work.
Details:
Spring boot: 2.2.7.RELEASE
Try this:
#TestPropertySource(properties = { "spring.config.location=classpath:foo.yml" })

Unable to generate different spring-cloud-gateway routes based on active Spring profile

I'm trying to define different routes in Spring Cloud Gateway using Spring profiles. Everything I've read about how Spring manages profiles using yaml sems to indicate it should work, but for the life of me it just ... doesn't seem to. (My other Spring apps use properties, so I'm unused to the yaml config -- it's possible I'm missing something.)
Effectively what I'm trying to do is to have a "prod" profile which contains the URIs for my production servers, and a "dev" profile which contains the localhost equivalents.
I have two profiles, dev and prod. My application.yml file looks like this
spring:
profiles:
default: prod
management:
endpoint:
health:
enabled: true
endpoints:
web:
exposure:
include: hystrix.stream, gateway
---
spring
profiles: prod
cloud:
gateway:
routes:
- id: test_route
uri: http://foo.mycompany.com
predicates:
- Path=/status
- Method=GET
---
spring
profiles: dev
cloud:
gateway:
routes:
- id: test_route
uri: http://localhost:8080
predicates:
- Path=/status
- Method=GET
My understanding is as follows:
the spring.profiles.default property tells Spring that, if no profile is specified, to use the prod profile
Spring will treat the --- as a "file separator" and re-evaluate each set of properties and overwrite previous values if the spring.profiles parameter evaluates true
Given this understanding, I would expect Spring to parse the "default" properties first, learning that the default activated profile should be prod. Then it will parse the prod properties. Since "prod" is an active profile (the only active profile, being the default), it should parse and apply the prod routes. Then it would parse the dev routes, but recognize that dev is not an active profile, and not overwrite those values. This is my understanding from reading the documentation on how to change config based on the environment.
However, when I load this, and I hit the actuator endpoint -- /actuator/gateway/routes -- I get back [] where I would expect to see the prod routes. I do see in my logs that the prod profile is activated, but it seems like not having the properties in the "default" section at top causes them to not be applied when the parser reads them out of the profile section.
The other thing I tried was putting the "dev" properties as the defaults, and then attempting to use the "prod" profile properties to overwrite the URIs. A similar issue happened there -- I hit the actuator endpoint and got back routes, but they were just the dev ones from the default.
How can I leverage Spring profiles to configure different Spring Cloud Config routes in my application.yml ?
Versions:
spring-cloud-gateway 2.0.1.BUILD-SNAPSHOT (to get a workaround for this bug, probably not relevant)
spring-cloud-starter-gateway
spring-boot 2.0.3.RELEASE
spring-boot-starter-webflux
spring-boot-starter-actuator
(I can't use Spring Cloud Config for political reasons. My company's chief architect has a severe case of Not Invented Here Syndrome.)
You cannot use spring.profiles.default in the property file. It will be too late for setting such value.
So you can set it using program argument (or System property). E.g.
java -jar --spring.profiles.default=dev your-app.jar
Or you can do it in the code by hardcoding the default profile:
#SpringBootApplication
public class DemoApplication {
public static void main(String[] args) {
SpringApplication application = new SpringApplication(DemoApplication.class);
ConfigurableEnvironment environment = new StandardEnvironment();
environment.setDefaultProfiles("dev");
application.setEnvironment(environment);
application.run(args);
}
}
Some related information you can find here by reading all comments:
https://github.com/spring-projects/spring-boot/issues/1219

Define Spring application properties default values on external library

In our company we have 100+ services built using Spring Boot. Most of these services share configuration values that allow us to configure how these applications register on Eureka, consume from Kafka and so on. This configuration is specified on the application.yml file in most of the services, and we'd like to avoid repeating these values, so we want to move this common configuration to a dependency that, when added to the classpath, will make that you no longer have to configure those values.
I achieved it by creating a new library which contains different #Configuration classes that configure the different beans with the right values. So now applications just need to add this library to the build.gradle file, and things are configured.
The problem is that, if one of the applications need to override any of the values for those beans, specifying the configuration in their application.yml file won't work, because I'm directly setting the right values to the bean.
For example, if my library declares this Bean
#Bean
#Profile("aws")
public EurekaInstanceConfigBean eurekaInstanceConfigAWS(InetUtils inetUtils) {
EurekaInstanceConfigBean eurekaInstanceConfigBean = new EurekaInstanceConfigBean(inetUtils);
AmazonInfo info = AmazonInfo.Builder.newBuilder().autoBuild("eureka");
eurekaInstanceConfigBean.setDataCenterInfo(info);
eurekaInstanceConfigBean.setInstanceId(info.get(AmazonInfo.MetaDataKey.instanceId));
eurekaInstanceConfigBean.setHostname(info.get(AmazonInfo.MetaDataKey.localHostname));
eurekaInstanceConfigBean.setIpAddress(info.get(AmazonInfo.MetaDataKey.localIpv4));
return eurekaInstanceConfigBean;
}
And someone specifies on the application.yml that they want a different hostname like
eureka:
instance:
hostname: some-custom-value.example.com
It will be ignored: the configuration set in the bean will be used, not the local application.yml values.
Is there a way to achieve an order of precedence here where values specified on application.yml are more important than the values on the bean coming from the library? I'd like a precedence like:
local application.yml > bean config in library > bean defaults coming from spring-boot/spring-cloud

How to load spring config xml ${ } values from spring boot application properties

Camel Spring Boot application version 2.20.1
How to load the application.properties values to spring config xml parameter on start up ?
For eg: few parameters are defined as follows :-
eas.ssl.key-store-type = JKS
eas.ssl.key-store-password = *****
eas.ssl.key-store = filelocation
Now the same need to be configured in Spring Config XML on startup as follows :
<sec:keyStore type="${eas.ssl.key-store-type}" password="${eas.ssl.key-store-password}" file="${eas.ssl.key-store}" />
In logs,I could see the properties from application properties are detected as expected
PropertySourcesPropertyResolver : Found key 'eas.ssl.key-store-password'
However it looks like on the keys are not applied to config xml as values.
The errors log states:-
org.apache.cxf.transport.https.SSLUtils : The key store password has not
been set via a system property or through configuration, reading data from
the keystore will fail.
As mentioned in my latest comment my primary issue was related to SSL HandshakeException (No appropraite protocol found). And I was working with JDK 1.8.
Later I realized it was cipher filter "SSL_ECDHE_RSA_WITH_AES_128_CBC_SHA256" was not supported with jdk 8. Once I changed to "TSL_ECDHE_RSA_WITH_AES_128_CBC_SHA256" all started working as expected.
A hint to solution got from this site :
http://www.ateam-oracle.com/tls-and-java/

Resources