Custom property loader with Spring Cloud Config - spring-boot

I'm using Spring Cloud Config in my spring-boot application and I need to write some custom code to handle properties to be read from my corporate password vault when property is flagged as such. I know spring cloud supports Hashicorp Vault, but that's not the one in case.
I don't want to hard-code specific properties to be retrieved from a different source, for example, I would have a properties file for application app1 with profile dev with values:
spring.datasource.url=jdbc:mysql://localhost/test
spring.datasource.username=dbuser
spring.datasource.password=dbpass
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
but for some other profiles such as prod, I would have:
spring.datasource.url=jdbc:mysql://localhost/test
spring.datasource.username=prod-user
spring.datasource.password=[[vault]]
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
So I need the custom property vault to intercept the property loaded whenever it finds a returned value equals to [[vault]] (or some other type of flag), and query from the corporate vault instead. In this case, my custom property loader would find the value of property spring.datasource.password from the corporate password vault. All other properties would still be returned as-is from values loaded by standard spring cloud config client.
I would like to do that using annotated code only, no XML configuration.

You can implement your own PropertySourceLocator and add entry to
spring.factories in directory META-INF.
#spring.factories
org.springframework.cloud.bootstrap.BootstrapConfiguration=/
foo.bar.MyPropertySourceLocator
Then you can you can refer to keys in your corporate password vault like a normal properties in spring.
spring.datasource.url=jdbc:mysql://localhost/test
spring.datasource.username=prod-user
spring.datasource.password=${lodaded.password.from.corporate.vault}
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
Implementation by HasiCorp: VaultPropertySourceLocatorSupport

While trying to solve the identical problem, I believe that I have come to work-around that may be acceptable.
Here is my solution below.
public class JBossVaultEnvironmentPostProcessor implements EnvironmentPostProcessor {
#Override
public void postProcessEnvironment(ConfigurableEnvironment environment, SpringApplication application) {
MutablePropertySources propertySources = environment.getPropertySources();
Map<String, String> sensitiveProperties = propertySources.stream()
.filter(propertySource -> propertySource instanceof EnumerablePropertySource)
.map(propertySource -> (EnumerablePropertySource<?>) propertySource)
.map(propertySource -> {
Map<String, String> vaultProperties = new HashMap<>();
String[] propertyNames = propertySource.getPropertyNames();
for (String propertyName : propertyNames) {
String propertyValue = propertySource.getProperty(propertyName).toString();
if (propertyValue.startsWith("VAULT::")) {
vaultProperties.put(propertyName, propertyValue);
}
}
return vaultProperties;
})
.reduce(new HashMap<>(), (m1, m2) -> {
m1.putAll(m2);
return m1;
});
Map<String, Object> vaultProperties = new HashMap<>();
sensitiveProperties.keySet().stream()
.forEach(key -> {
vaultProperties.put(key, VaultReader.readAttributeValue(sensitiveProperties.get(key)));
});
propertySources.addFirst(new MapPropertySource("vaultProperties", vaultProperties));
}

Related

Register MetadataBuilderContributor based on database type in Spring Boot

I added a few implementations of MetadataBuilderContributor based on the database (h2, mysql, oracle) since they all have a slightly different syntax.
As of now, registering of the contributors works through the property in application.yml:
spring:
jpa:
properties:
hibernate:
metadata_builder_contributor: org.foo.bar.H2Implementation
I am aware that I can create multiple profiles -h2, -mysql, -oracle to apply the correct contributors. However, I'd like to automatically set these based on the driverClassName that was set (if I can find a match, otherwise default to the application.yml)
Is there a way to do this and not require the entry in my application.yml?
Here is my solution that Frame91 mentioned, using a Spring ApplicationEnvironmentPreparedEvent
public class MetadataBuilderContributerResolver
implements ApplicationListener<ApplicationEnvironmentPreparedEvent>
{
#Override
public void onApplicationEvent(ApplicationEnvironmentPreparedEvent event) {
ConfigurableEnvironment environment = event.getEnvironment();
String driverClassName = environment.getProperty("spring.datasource.driverClassName");
Class<?> metadataBuilderContributorClazz = switch (driverClassName) {
case "org.h2.Driver" -> H2MetadataBuilderContributor.class;
case "org.mariadb.jdbc.Driver" -> MariaDbMetadataBuilderContributor.class;
case "oracle.jdbc.OracleDriver" -> OracleMetadataBuilderContributor.class;
default -> throw new IllegalArgumentException("Unsupported driver: " + driverClassName);
};
String className = metadataBuilderContributorClazz.getName();
Properties props = new Properties();
props.put("spring.jpa.properties.hibernate.metadata_builder_contributor", className);
environment.getPropertySources().addFirst(new PropertiesPropertySource(this.getClass().getName(), props));
}
}
Thanks to Frame91

List distributed properties in Spring Boot

Code sample below is from a Spring Boot application that obtains property definitions from the local application.properties file. Additional or overriding property definitions are obtained from a remote Consul server. The goal is to have the application enumerate all property definitions regardless of the source. Example setup:
The local application.properties file contains this property definition
spring.cloud.stream.bindings.breakoutOutputChannel.destination=breakout_topic_local
The overriding definition for the same property in the Consul server is:
spring.cloud.stream.bindings.breakoutOutputChannel.destination=breakout_topic_production
In the same code I am using two mechanisms to access the same property value. First is use of the #Value annotation, the second is querying properties obtained from the environment. The means of obtaining the applications enviroment properties is also show below.
When using the #Value annotation the proper override value is obtained. But when accessing the property definition directly only the value from the local file is available. Example show below.
Is it possible to obtain the same Map of properties that the #Value annotation uses so that its contents can be enumerated?
Sample code:
#Value("${spring.cloud.stream.bindings.breakoutOutputChannel.destination}")
String breakout;
#EventListener
public void onApplicationEvent(ApplicationReadyEvent event) {
Properties props =applicationProperties(event.getApplicationContext().getEnvironment());
log.info("Property spring.cloud.stream.bindings.breakoutOutputChannel.destination");
log.info( " value from bound var = " +breakout);
log.info( " value from properties = " +props.getProperty("spring.cloud.stream.bindings.breakoutOutputChannel.destination"));
}
public Properties applicationProperties(Environment env) {
final Properties properties = new Properties();
for(Iterator it = ((AbstractEnvironment) env).getPropertySources().iterator(); it.hasNext(); ) {
PropertySource propertySource = (PropertySource) it.next();
if (propertySource instanceof PropertiesPropertySource) {
log.info("Adding all properties contained in " + propertySource.getName());
properties.putAll(((MapPropertySource) propertySource).getSource());
}
if (propertySource instanceof CompositePropertySource){
properties.putAll(getPropertiesInCompositePropertySource((CompositePropertySource) propertySource));
}
}
return properties;
}
private Properties getPropertiesInCompositePropertySource(CompositePropertySource compositePropertySource){
final Properties properties = new Properties();
compositePropertySource.getPropertySources().forEach(propertySource -> {
if (propertySource instanceof MapPropertySource) {
log.info("Adding all properties contained in " + propertySource.getName());
properties.putAll(((MapPropertySource) propertySource).getSource());
}
if (propertySource instanceof CompositePropertySource)
properties.putAll(getPropertiesInCompositePropertySource((CompositePropertySource) propertySource));
});
return properties;
}
Sample output
Adding all properties contained in applicationConfig: [file:./application.properties]
Adding all properties contained in applicationConfig: [classpath:/application.properties]
Property spring.cloud.stream.bindings.breakoutOutputChannel.destination
value from bound var = breakout_topic_production
value from properties = breakout_topic_local
While the generated properties object, created in the above example, source is the application Environment, the values are indeed only those that are included in the local property definition files.
However if the property key is used to get the value directly from the Environment, the the value returned will contain any override distributed to the application from Consul. From the example the return value from
event.getApplicationContext().getEnvironment().getProperty("spring.cloud.stream.bindings.breakoutOutputChannel.destination"));
will return the correct property value.

How to make externalized properties file available to spring boot integration tests?

I am developing an application using Spring Boot. I have an externalized properties file on file system. Its location is stored in an environment variable as below--
export props=file://Path-to-file-on-filesystem/file.properties
Properties from this file are loaded on classpath and are made available to application like below--
List<String> argList = new ArrayList<>();
String properties = System.getenv().get("props");
try (InputStream is = BinaryFileReaderImpl.getInstance().getResourceAsStream(properties)) {
if (is != null) {
Properties props = new Properties();
props.load(is);
for(String prop : props.stringPropertyNames()) {
argList.add("--" + prop + "=" + props.getProperty(prop));
}
}
} catch(Exception e) {
//exception handling
}
This argList is passed to SpringBootApplication when it starts like below--
SpringApplication.run(MainApplication.class, argList);
I can access all the properties using ${prop.name}
However, I do not have access to these properties when I run JUnit Integration Tests. All my DB properties are in this externalized properties file. I do not want to keep this file anywhere in the application eg. src/main/resources
Is there any way I can load these properties in spring's test context?
I could finally read the externalized Properties file using #PropertySource on linux machine. Could not get it working on Windows instance though.
Changes done are as below-
export props=/path-to-file
Note that file:// and actual file name has been removed from environment variable.
#Configuration
#PropertySource({"file:${props}/file-test.properties", "classpath:some_other_file.properties"})
public class TestConfiguration {
}
Keep this configuration class in individual projects' src/test/java folder. Thank you Joe Chiavaroli for your comment above.

Dump Spring boot Configuration

Our Ops guys want the Spring boot configuration (i.e. all properties) to be dumped to the log file when the app starts. I assume this can be done by injecting the properties with annotation #ConfigurationProperties and printing them.
The questions is whether there is a better or built-in mechanism to achieve this.
Given there does not seem to be a built in solution besides, I was try to cook my own. Here is what I came up with:
#Component
public class ConfigurationDumper {
#Autowired
public void init(Environment env){
log.info("{}",env);
}
}
The challenge with this is that it does not print variables that are in my application.yml. Instead, here is what I get:
StandardServletEnvironment
{
activeProfiles=[],
defaultProfiles=[default],
propertySources=[
servletConfigInitParams,
servletContextInitParams,
systemProperties,
systemEnvironment,
random,
applicationConfig: [classpath: /application.yml]
]
}
How can I fix this so as to have all properties loaded and printed?
If you use actuator , env endpoint will give you all the configuration properties set in ConfigurableEnvironment and configprops will give you the list of #ConfigurationProperties, but not in the log.
Take a look at the source code for this env endpoint, may be it will give you an idea of how you could get all the properties you are interested in.
There is no built-in mechanism and it really depends what you mean by "all properties". Do you want only the actual keys that you wrote or you want all properties (including defaults).
For the former, you could easily listen for ApplicationEnvironmentPreparedEvent and log the property sources you're interested in. For the latter, /configprops is indeed a much better/complete output.
This logs only the properties configured *.properties file.
/**
* maps given property names to its origin
* #return a map where key is property name and value the origin
*/
public Map<String, String> fromWhere() {
final Map<String, String> mapToLog = new HashMap<>();
final MutablePropertySources propertySources = env.getPropertySources();
final Iterator<?> it = propertySources.iterator();
while (it.hasNext()) {
final Object object = it.next();
if (object instanceof MapPropertySource) {
MapPropertySource propertySource = (MapPropertySource) object;
String propertySourceName = propertySource.getName();
if (propertySourceName.contains("properties")) {
Map<String, Object> sourceMap = propertySource.getSource();
for (String key : sourceMap.keySet()) {
final String envValue = env.getProperty(key);
String env2Val = System.getProperty(key);
String source = propertySource.getName().contains("file:") ? "FILE" : "JAR";
if (envValue.equals(env2Val)) {
source = "ENV";
}
mapToLog.putIfAbsent(key, source);
}
}
}
}
return mapToLog;
}
my example output which depicts the property name, value and from where it comes. My property values are describing from where they come.:
myprop: fooFromJar from JAR
aPropFromFile: fromExternalConfFile from FILE
mypropEnv: here from vm arg from ENV
ENV means that I have given it by -D to JVM.
JAR means it is from application.properties inside JAR
FILE means it is from application.properties outside JAR

PropertySourcesPlaceholderConfigurer not registering with Environment in a SpringBoot Project

I am moving a working project from using SpringBoot command line arguments to reading properties from a file. Here are the involved portions of the #Configuration class:
#Configuration
class RemoteCommunication {
#Inject
StandardServletEnvironment env
#Bean
static PropertySourcesPlaceholderConfigurer placeholderConfigurer () {
// VERIFIED this is executing...
PropertySourcesPlaceholderConfigurer target = new PropertySourcesPlaceholderConfigurer()
// VERIFIED this files exists, is readable, is a valid properties file
target.setLocation (new FileSystemResource ('/Users/me/Desktop/mess.properties'))
// A Debugger does NOT show this property source in the inject Environment
target
}
#Bean // There are many of these for different services, only one shown here.
MedicalSorIdService medicalSorIdService () {
serviceInstantiator (MedicalSorIdService_EpicSoap, 'uri.sor.id.lookup.internal')
}
// HELPER METHODS...
private <T> T serviceInstantiator (final Class<T> classToInstantiate, final String propertyKeyPrimary) {
def value = retrieveSpringPropertyFromConfigurationParameter (propertyKeyPrimary)
classToInstantiate.newInstance (value)
}
private def retrieveSpringPropertyFromConfigurationParameter (String propertyKeyPrimary) {
// PROBLEM: the property is not found in the Environment
def value = env.getProperty (propertyKeyPrimary, '')
if (value.isEmpty ()) throw new IllegalStateException ('Missing configuration parameter: ' + "\"$propertyKeyPrimary\"")
value
}
Using #Value to inject the properties does work, however I'd rather work with the Environment directly if at all possible. If the settings are not in the Environment then I am not exactly sure where #Value is pulling them from...
env.getProperty() continues to work well when I pass in command line arguments specifying the properties though.
Any suggestions are welcome!
The issue here is the distinction between PropertySourcesPlaceholderConfigurer and StandardServletEnvironment, or Environment for simplicity.
The Environment is an object that backs the whole ApplicationContext and can resolve a bunch of properties (the Environment interface extends PropertyResolver). A ConfigurableEnvironment has a MutablePropertySources object which you can retrieve through getPropertySources(). This MutablePropertySources holds a LinkedList of PropertySource objects which are checked in order to resolve a requested property.
PropertySourcesPlaceholderConfigurer is a separate object with its own state. It holds its own MutablePropertySources object for resolving property placeholders. PropertySourcesPlaceholderConfigurer implements EnvironmentAware so when the ApplicationContext gets hold of it, it gives it its Environment object. The PropertySourcesPlaceholderConfigurer adds this Environment's MutablePropertySources to its own. It then also adds the various Resource objects you specified with setLocation() as additional properties. These Resource objects are not added to the Environment's MutablePropertySources and therefore aren't available with env.getProperty(String).
So you cannot get the properties loaded by the PropertySourcesPlaceholderConfigurer into the Environment directly. What you can do instead is add directly to the Environment's MutablePropertySouces. One way is with
#PostConstruct
public void setup() throws IOException {
Resource resource = new FileSystemResource("spring.properties"); // your file
Properties result = new Properties();
PropertiesLoaderUtils.fillProperties(result, resource);
env.getPropertySources().addLast(new PropertiesPropertySource("custom", result));
}
or simply (thanks #M.Deinum)
#PostConstruct
public void setup() throws IOException {
env.getPropertySources().addLast(new ResourcePropertySource("custom", "file:spring.properties")); // the name 'custom' can come from anywhere
}
Note that adding a #PropertySource has the same effect, ie. adding directly to the Environment, but you're doing it statically rather than dynamically.
In SpringBoot it's enough to use #EnableConfigurationProperties annotation - you don't need to setup PropertySourcesPlaceholderConfigurer.
Then on POJO you add annotation #ConfigurationProperties and Spring automatically injects your properties defined in application.properties.
You can also use YAML files - you just need to add proper dependency (like SnakeYaml) to classpath
You can find detailed example here: http://spring.io/blog/2013/10/30/empowering-your-apps-with-spring-boot-s-property-support
I achieved this during PropertySourcesPlaceholderConfigurer instantiation.
#Bean
public static PropertySourcesPlaceholderConfigurer propertySourcesPlaceholderConfigurerBean(Environment env) {
PropertySourcesPlaceholderConfigurer propertySourcesPlaceholderConfigurer = new PropertySourcesPlaceholderConfigurer();
YamlPropertiesFactoryBean yamlFactorybean = new YamlPropertiesFactoryBean();
yamlFactorybean.setResources(determineResources(env));
PropertiesPropertySource yampProperties = new PropertiesPropertySource("yml", yamlFactorybean.getObject());
((AbstractEnvironment)env).getPropertySources().addLast(yampProperties);
propertySourcesPlaceholderConfigurer.setProperties(yamlFactorybean.getObject());
return propertySourcesPlaceholderConfigurer;
}
private static Resource[] determineResources(Environment env){
int numberOfActiveProfiles = env.getActiveProfiles().length;
ArrayList<Resource> properties = new ArrayList(numberOfActiveProfiles);
properties.add( new ClassPathResource("application.yml") );
for (String profile : env.getActiveProfiles()){
String yamlFile = "application-"+profile+".yml";
ClassPathResource props = new ClassPathResource(yamlFile);
if (!props.exists()){
log.info("Configuration file {} for profile {} does not exist");
continue;
}
properties.add(props);
}
if (log.isDebugEnabled())
log.debug("Populating application context with properties files: {}", properties);
return properties.toArray(new Resource[properties.size()]);
}
Maybe all you need is to set -Dspring.config.location=... (alternatively SPRING_CONFIG_LOCATION as an env var)? That has the effect of adding an additional config file to the default path for the app at runtime which takes precedence over the normal application.properties? See howto docs for details.

Resources