I have a YAML configuration file with a map of properties:
properties:
a.b.c: 1
Boot will parse this as:
{a:{b:{c:1}}}
However, what I desire is :
{'a.b.c': 1}
Is there anyway to coax it into "pass through" key mode? Quoting the key doesn't seem to help.
Update
Actual example below.
Java
import static com.google.common.collect.Maps.newLinkedHashMap;
import java.util.Map;
import javax.annotation.PostConstruct;
import lombok.Data;
import lombok.val;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Configuration;
#Data
#Configuration
#ConfigurationProperties("hadoop")
public class HadoopProperties {
private Map<Object, Object> properties = newLinkedHashMap();
}
YAML
application.yml:
hadoop:
properties:
fs.defaultFS: hdfs://localhost:8020
mapred.job.tracker: localhost:8021
Result
Calling toString() on the resulting object:
HadoopProperties(properties={fs={defaultFS=hdfs://localhost:8020}, mapred={job={tracker=localhost:8021}}})
I see. It's because you are binding to a very generic object, so Spring Boot thinks your period separators are map key dereferences. What happens if you bind to Map or Properties?
you can use as below
hadoop:
properties:
"[fs.defaultFS]" : hdfs://localhost:8020
"[mapred.job.tracker]": localhost:8021
reference - link
Related
I have migrated an extension from quarkus 2.7.5 to quarkus 2.8.0.
After the migration, I run mvn clean install and the console shows me weird warnings about all (maybe 100) config properties (some of them are defined by me, other not like java.specification.version):
[WARNING] [io.quarkus.config] Unrecognized configuration key "my.property" was provided; it will be ignored; verify that the dependency extension for this configuration is set or that you did not make a typo
I think my integration-tests module causes this issue.
Here is my class in runtime folder:
import java.util.Optional;
import io.quarkus.runtime.annotations.ConfigItem;
import io.quarkus.runtime.annotations.ConfigPhase;
import io.quarkus.runtime.annotations.ConfigRoot;
#ConfigRoot(phase = ConfigPhase.RUN_TIME, prefix="", name = "myApp")
public class MyAppConfig {
#ConfigItem(defaultValue = "defaultValue")
String firstProperty;
#ConfigItem(defaultValue = "")
Optional<String> secondProperty;
#ConfigItem(defaultValue = "defaultValue")
String thirdProperty;
// Getters ...
}
Here is my test:
import static org.junit.jupiter.api.Assertions.assertEquals;
import org.eclipse.microprofile.config.inject.ConfigProperty;
import org.junit.jupiter.api.Test;
import io.quarkus.test.junit.QuarkusTest;
#QuarkusTest
public class MyAppIntegrationTest {
#ConfigProperty(name="myApp.first-property")
String firstProperty;
#ConfigProperty(name="myApp.second-property")
String secondProperty;
#ConfigProperty(name="myApp.third-property")
String thirdProperty;
#Test
public void testConfig() {
assertEquals("firstValue", firstProperty);
assertEquals("secondValue", secondProperty);
assertEquals("thirdValue", thirdProperty);
}
}
Can someone help me on this ? For instance, do I need a BuildItem for that ?
Thanks for your help
After spending hours on those warnings, I found that they was caused by the empty prefix field of the ConfigRoot annotation.
Setting name field to "" and prefix to "myApp" fixed the issue
I have mounted one volume which contained username and password inside pod. If I do:
kubectl exec -it my-app -- cat /mnt/secrets-store/git-token
{"USERNAME":"usernameofgit","PASSWORD":"dhdhfhehfhel"}
I want to read this USERNAME and PASSWORD using Spring Boot.
Assuming:
the file (git_token) format is fixed (JSON).
the file may not have an extension suffix (.json).
... we have some Problems!
I tried 2.3.5. Importing Extensionless Files like:
spring.config.import=/mnt/secrets-store/git-token[.json]
But it works only with YAML/.properties yet!(tested with spring-boot:2.6.1))
Same applies to 2.8. Type-safe Configuration Properties. ;(;(
In Spring-Boot we can (out-of-the box) provide JSON-config (only) as SPRING_APPLICATION_JSON environment/command line property, and it has to be the json string, and cannot be a path or file (yet).
The proposed (baeldung) article shows ways to "enable JSON properties", but it is a long article with many details, shows much code and has decent lacks/outdates (#Component on #ConfigurationProperties is rather "unconventional")..
I tried the following (on local machine, under the mentioned assumptions):
package com.example.demo;
import com.fasterxml.jackson.annotation.JsonProperty;
import lombok.Data;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
#SpringBootApplication
public class DemoApplication {
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
#Value("""
#{#jacksonObjectMapper.readValue(
T(java.nio.file.Files).newInputStream(
T(java.nio.file.Path).of('/mnt/secrets-store/git-token')),
T(com.example.demo.GitInfo)
)}""" // watch out with #Value and text blocks! (otherwise: No converter found capable of converting from type [com.example.demo.GitInfo] to type [java.lang.String])
)
GitInfo gitInfo;
#Bean
CommandLineRunner runner() {
return (String... args) -> {
System.out.println(gitInfo.getUsername());
System.out.println(gitInfo.getPassword());
};
}
}
#Data
class GitInfo {
#JsonProperty("USERNAME")
private String username;
#JsonProperty("PASSWORD")
private String password;
}
With (only) spring-boot-starter-web and lombok on board, it prints the expected output.
Solution outline:
a pojo for this
the upper case is little problematic, but can be handled as shown.
a (crazy) #Value - (Spring-)Expression, involving:
(hopefully) auto-configured #jacksonObjectMapper bean. (alternatively: custom)
ObjectMapper#readValue (alternatives possible)
java.nio.file.Files#newInputStream (alternatives possible)
java.nio.file.Path#of
When you have your volume mounted, then all you need to do is to read a JSON file from the Spring Boot application. I recommend reading Load Spring Boot Properties From a JSON File.
In short, you can create a class corresponding to your JSON file, something like this one.
#Component
#PropertySource("file:/mnt/secrets-store/git-token")
#ConfigurationProperties
public class GitToken {
private String username;
private String password;
// getters and setters
}
Then, you need to add it to componentScan and you can autowire your class.
Hi I am new to Spring Boot (but have been using Spring in my apps for a while). I am trying to use a custom SSM PropertyPlaceholderConfigurer, based on my SSM Client, that reads my properties from AWS SSM, in addition to properties in my normal application.properties.
This code works fine in my pre-spring-boot application. However, in the new application, I see that it overrides the application.properties. And this seems to be a well documented problem.
So i decided to include the application.properties file in my custom PropertyPlaceholderConfigurer class and load all the properties together and still it does not resolve any properties in application.properties that are marked with "${}" and resolve by my custom location. What more do i need to do?
As an alternative, I tried to have the properties i need to load from SSM to be loaded via an EnvironmentPostProcessor but it was unable to connect to the AWS SSM server at this point in the loading process (not sure why)
The answer is to use the EnvironmentPostProcessor. Works perfectly. See code below:
import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.env.EnvironmentPostProcessor;
import org.springframework.core.env.ConfigurableEnvironment;
import org.springframework.core.env.MapPropertySource;
import java.util.HashMap;
import java.util.Map;
/**
* This class loads SSM parameters base on region and environment
* Needs to be added to spring.factories class so that it will be invoked. as follow:
* org.springframework.boot.env.EnvironmentPostProcessor=<full package>.SSMEnvironmentPostProcessor
* Add the SSM propeties to other properties already set
*/
public class SSMEnvironmentPostProcessor implements EnvironmentPostProcessor {
private static final String QUOTE = "\"";
#Override
public void postProcessEnvironment(ConfigurableEnvironment environment, SpringApplication application) {
SSMClient ssmClient = new SSMClient(DefaultAWSCredentialsProviderChain.getInstance(), System.getProperty("env" +
".region"), new ClientConfiguration());
ssmClient.init();
Map<String, Object> parameters = new HashMap<>();
ssmClient.getParametersByPath("/" + System.getProperty("env"), true).entrySet().stream()
.forEach(entry -> parameters.put(entry.getKey(), entry.getValue()));
MapPropertySource mapPropertySource = new MapPropertySource("ssm", parameters);
environment.getPropertySources().addLast(mapPropertySource);
}
}
I have a number of non spring-boot services which generate JVM metrics automatically using io.prometheus simpleclient_hotspot.
I am trying to obtain parity with these and the JVM metrics produced from my spring-boot services which are using micrometer's default settings.
Is there some additional configuration for micrometer which I need in order for it to generate metrics such as:
# TYPE jvm_info gauge
jvm_info{version="
...
jvm_memory_bytes_used
jvm_memory_pool_bytes_max
etc...
?
I appreciate micrometer out of the box is logging: jvm_memory_used_bytes which doesn't match the name of the same metric from simpleclient_hotspot :(
I would like to achieve consistency if possible, and jvm_info would be particularly useful from micrometer/spring-boot.
I recommend you look into NamingConvention found in the micrometer core library. There are several examples of using it to convert names to different monitoring systems (look for classes implenting NamingConvention). For example, if you're using Prometheus (which it looks like you might be), you could look at PrometheusNamingConvention as an example. It looks like you'll be interested in implementing/overriding the public String name(String name, Meter.Type type, #Nullable String baseUnit) method.
To answer my own question(s)
I ended up implementing this for JVM Version Info:
import io.micrometer.core.instrument.Gauge;
import io.micrometer.core.instrument.MeterRegistry;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;
/**
* Migrated From simpleclient_hotspot/src/main/java/io/prometheus/client/hotspot/VersionInfoExports.java
* Because its not included in Micrometer.
*/
#Component
public class JvmVersionInfoExports {
private static final Logger LOGGER = LoggerFactory.getLogger(JvmVersionInfoExports.class);
public JvmVersionInfoExports(MeterRegistry meterRegistry) {
LOGGER.info("Adding JVM Metrics");
Gauge.builder("jvm_info", () -> 1L)
.description("JVM version info")
.tag("version", System.getProperty("java.runtime.version", "unknown"))
.tag("vendor", System.getProperty("java.vm.vendor", "unknown"))
.tag("runtime", System.getProperty("java.runtime.name", "unknown"))
.register(meterRegistry);
}
}
I ended up implementing this for OS Version Info:
import io.micrometer.core.instrument.Gauge;
import io.micrometer.core.instrument.MeterRegistry;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;
#Component
public class OsVersionInfoExports {
private static final Logger LOGGER = LoggerFactory.getLogger(OsVersionInfoExports.class);
public OsVersionInfoExports(MeterRegistry meterRegistry) {
LOGGER.info("Adding OS Metrics");
Gauge.builder("os_info", () -> 1L)
.description("OS version info")
.tag("version", System.getProperty("os.version", "unknown"))
.tag("arch", System.getProperty("os.arch", "unknown"))
.tag("name", System.getProperty("os.name", "unknown"))
.register(meterRegistry);
}
}
Although theses misuse the Micrometer Gauge they provide me with the compatibility I need to track these details across services using a mix of Micrometer and Prometheus HotSpot libraries.
I configured Spring Cloud Config server like this:
#SpringBootApplication
#EnableAutoConfiguration
#EnableConfigServer
public class ConfigServer {
public static void main(String[] args) {
SpringApplication.run(ConfigServer.class, args);
}
}
I'm using 'native' profile so properties are picked up from the file system:
server.port=8888
spring.profiles.active=native
spring.cloud.config.server.native.search-locations: classpath:/global
Now the tricky part is that some properties contain environmental variables. Properties in 'global/application-production.properties' are configured like this:
test=${DOCKER_HOST}
When I start up Config Server - everything works fine. However when I access http://localhost:8888/testapp/production I see this:
{
name: "testapp",
profiles: [
"production"
],
label: null,
version: null,
propertySources: [
{
name: "classpath:/global/application-production.properties",
source: {
test: "${DOCKER_HOST}"
}
}
]
}
So value from ENV variable is not replacing ${DOCKER_HOST} put rather returned as is.
But if I access http://localhost:8888/application-production.properties then result is non JSON but rather plain text:
test: tcp://192.168.99.100:2376
Spring documentation says:
The YAML and properties representations have an additional flag (provided as a boolean query parameter resolvePlaceholders) to signal that placeholders in the source documents, in the standard Spring ${…} form, should be resolved in the output where possible before rendering. This is a useful feature for consumers that don’t know about the Spring placeholder conventions.
For some reason resolvePlaceholders is not working for JSON representation thus server config clients need to be aware of all ENV variables configured on server.
Is it possible to force JSON representation resolvePlaceholders same way as plain text (properties) representation?
I faced the same issue. After looking into Spring Cloud Config Repository I have found the following commit:
Omit system properties and env vars from placeholders in config
It looks like such behavior is not supported.
You can try the Property Overrides feature to override properties from git Environment Repository.
To override property foo at runtime, just set a system property or an environment variable spring.cloud.config.server.overrides.foo before starting the config server.
There was an update in order to accomplish this, in the following merge. 1 I found an implementation for resolvePlaceholders. Which gave me the idea of just creating a new controller which uses the EnvironmentController. This will allow you to resolve configuration, this is a good bootstrap.
import com.fasterxml.jackson.databind.ObjectMapper;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.cloud.config.server.environment.EnvironmentController;
import org.springframework.cloud.config.server.environment.EnvironmentRepository;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;
#RestController
#RequestMapping(method = RequestMethod.GET, path = "resolved/${spring.cloud.config.server.prefix:}")
public class ReplacedEnvironmentController {
private EnvironmentController environmentController;
#Autowired
public ReplacedEnvironmentController(EnvironmentRepository repository) {
environmentController = new EnvironmentController(repository, new ObjectMapper());
}
public ReplacedEnvironmentController(EnvironmentRepository repository,
ObjectMapper objectMapper) {
environmentController = new EnvironmentController(repository, objectMapper);
}
#RequestMapping("/{name}/{profiles:.*[^-].*}")
public ResponseEntity<String> resolvedDefaultLabel(#PathVariable String name,
#PathVariable String profiles) throws Exception {
return resolvedLabelled(name, profiles, null);
}
#RequestMapping("/{name}/{profiles}/{label:.*}")
public ResponseEntity<String> resolvedLabelled(#PathVariable String name, #PathVariable String profiles,
#PathVariable String label) throws Exception {
return environmentController.labelledJsonProperties(name, profiles, label, true);
}
}