Spring Cloud Kubernetes not reloading secret change - spring-boot

I am exploring the capabilities of Spring Cloud Kubernetes by using its feature to reload secrets on the fly. However, I still did not get this working.
I have a simple Spring Boot application that just prints out the contents of the secret that is mounted in the pod. Here is the configuration in bootstrap.properties
spring.cloud.kubernetes.reload.enabled=true
spring.cloud.kubernetes.reload.monitoring-secrets=true
spring.cloud.kubernetes.secrets.enabled=true
spring.cloud.kubernetes.secrets.paths=/etc/secret-volume
management.endpoint.info.enabled=true
management.endpoint.health.enabled=true
management.endpoint.restart.enabled=true
In application.properties, I have defined the property to get the value of the secret:
mysecret.password=${MY-PWD}
In the Spring Boot application, I defined a bean that will store the value of the secret:
#Configuration
#ConfigurationProperties(prefix = "mysecret")
public class MySecret {
private String password;
public String getPassword() {
return password;
}
public void setPassword(String password) {
this.password = password;
}
}
When I run the application on minikube, I see in the logs that Spring detects the declared secret and activates the profile:
16:54:30.887 [main] INFO
o.s.c.b.c.PropertySourceBootstrapConfiguration - Located property
source: [BootstrapPropertySource#1132379993
{name='bootstrapProperties-my-pwd', properties={MY-PWD=qwerty}}]
16:54:30.899 [main] INFO c.c.r.ReloadSecretsApplication - The
following profiles are active: kubernetes
After a while, I get the following log that says that it was added a watcher over the secret:
16:54:35.460 [OkHttp https://10.96.0.1/...] DEBUG
i.f.k.c.d.i.WatchConnectionManager - WebSocket successfully opened
16:54:35.460 [main] INFO
o.s.c.k.c.r.EventBasedConfigurationChangeDetector - Added new
Kubernetes watch: secrets-watch 16:54:35.460 [main] INFO
o.s.c.k.c.r.EventBasedConfigurationChangeDetector - Kubernetes
event-based configuration change detector activated
Then, when I change the secret, I get this line saying that the reload won't be triggered:
11:20:15.963 [OkHttp https://10.96.0.1/...] WARN
o.s.c.k.c.r.EventBasedConfigurationChangeDetector - The current number
of Confimap PropertySources does not match the ones loaded from the
Kubernetes - No reload will take place
The documentation is very scarce about this topic.
Do I have any missing configuration here?
Link to the Spring Boot Application: https://github.com/Azlop/spring-cloud-kubernetes-reload-secrets

I know this is an old question without answer. I met the same issue and solved it after some research. Here I share my experience on this question if it can help someone in the future.
Spring Cloud Kubernetes has stored the external configmaps as several ConfigMapPropertySource instances in a CompositePropertySource or BootstrapPropertySource. When the application minitors the configmaps have been changed, it takes the ConfigMapPropertySource instances from the CompositePropertySource and compares the length of the ConfigMapPropertySource instances with the incoming instances. If the lengths are not same, the application returns an error message: "The current number of Confimap PropertySources does not match the ones loaded from the Kubernetes - No reload will take place".
But here is a potential problem: The other functions, in my case it is an encryptor, may change the type of the CompositePropertySource. Later, the application can't get the ConfigMapPropertySource instances and the comparison must be failed.
So my solution is to create a customized EventBasedConfigurationChangeDetector which has been changed the logic of the comparison. The original version is:
private void onEvent(ConfigMap configMap) {
boolean changed = changed(locateMapPropertySources(this.configMapPropertySourceLocator, this.environment), findPropertySources(ConfigMapPropertySource.class));
if (changed) {
XXXX
}
}
You should rewrite the findPropertySources function to fit in your stituation. In my instance, I unwrap the encrypted PropertySource to obtain the original ConfigMapPropertySource instances.
Finally, injecting your customized EventBasedConfigurationChangeDetector to replace the vanilla one will resolve the issue.

Related

Embedded Keycloak springboot application, upgrading keycloak from 12 to 14

Hello I have an SpringBoot application with embedded Keyclock created using the below example
https://www.baeldung.com/keycloak-embedded-in-spring-boot-app
I am trying to upgrade Keycloak from 12 to 14 and also upgraded resteasy to 3.15.1.Final, infinispan to 11.0.9.Final. The application fails to start and there is not much in the logs except for a NullPointerException. I am trying to see if any one tried the same and has any suggestions as there are not much in logs. Thanks
2021-07-07 10:02:05 [main] INFO org.keycloak.services - KC-SERVICES0050: Initializing master realm
2021-07-07 10:02:07 [main] DEBUG org.keycloak.keys.GeneratedRsaKeyProviderFactory - Generated keys for master
2021-07-07 10:02:07 [main] DEBUG org.keycloak.keys.GeneratedHmacKeyProviderFactory - Generated secret for master
2021-07-07 10:02:07 [main] DEBUG org.keycloak.keys.GeneratedAesKeyProviderFactory - Generated secret for master
2021-07-07 10:02:07 [main] DEBUG org.keycloak.connections.jpa.updater.liquibase.lock.LiquibaseDBLockProvider - Going to release database lock namespace=KEYCLOAK_BOOT
2021-07-07 10:02:07 [main] DEBUG org.keycloak.connections.jpa.updater.liquibase.lock.CustomLockService - Going to release database lock
2021-07-07 10:02:07 [main] DEBUG org.keycloak.models.cache.infinispan.InfinispanUserCacheProviderFactory - Registered cluster listeners
2021-07-07 10:02:07 [main] DEBUG org.keycloak.executors.DefaultExecutorsProviderFactory - We are in managed environment. Executor 'java:jboss/ee/concurrency/executor/default' was available.
2021-07-07 10:02:07 [main] DEBUG org.keycloak.executors.DefaultExecutorsProviderFactory - Found executor for 'storage-provider-threads' under JNDI name 'java:jboss/ee/concurrency/executor/storage-provider-threads'
2021-07-07 10:02:07 [main] FATAL org.keycloak.services - java.lang.NullPointerException
2021-07-07 10:02:07 [main] DEBUG org.keycloak.executors.DefaultExecutorsProviderFactory - Found executor for 'storage-provider-threads' under JNDI name 'java:jboss/ee/concurrency/executor/storage-provider-threads'
$$$$$$$$$$$$$$$$Stachtrace$$$$$$$$$$$$$ : java.lang.NullPointerException
at java.base/java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
at java.base/java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
at org.keycloak.executors.DefaultExecutorsProviderFactory.getExecutor(DefaultExecutorsProviderFactory.java:113)
at org.keycloak.executors.DefaultExecutorsProviderFactory$1.getExecutor(DefaultExecutorsProviderFactory.java:68)
at org.keycloak.utils.ServicesUtils.timeBoundOne(ServicesUtils.java:71)
at org.keycloak.storage.AbstractStorageManager.mapEnabledStorageProvidersWithTimeout(AbstractStorageManager.java:135)
at org.keycloak.storage.UserStorageManager.getUsersCount(UserStorageManager.java:374)
at org.keycloak.models.cache.infinispan.UserCacheSession.getUsersCount(UserCacheSession.java:545)
at org.keycloak.storage.user.UserQueryProvider.getUsersCount(UserQueryProvider.java:52)
at org.keycloak.services.managers.ApplianceBootstrap.createMasterRealmUser(ApplianceBootstrap.java:99)
We had similar vague error messages when upgrading our embedded keycloak to 15 in our Spring Boot/Servlet-style application. But we eventually fixed them by crawling through the code/history of the project that this baeldung.com tutorial was based; thomasdarimont's Embedded Keycloak Server running in a Spring Boot App. This repo is a few upgrades past the initial tutorial; up to Keycloak 13. We compared the history to our code and applying/testing fixes incrementally for each version from keycloak 11 to 13. After that, the updates from 13 to 15 were a bit easier to understand, and the errors were not as cryptic.
Our application is not exactly like the example code given in the Baeldung tutorial, but below is a list of changes we needed to upgrade to Keycloak 15.0.1. These changes may not be 1:1 with your project, but hopefully these help someone save some time.
In KeycloakApplication moved the realm-setup functions from constructor method to bootstrap() method. (An example fix found here)
public class EmbeddedKeycloakApplication extends KeycloakApplication {
...
//public EmbeddedKeycloakApplication() {
// createMasterRealmAdminUser();
// createBaeldungRealm();
//}
#Override
protected ExportImportManager bootstrap() {
final ExportImportManager exportImportManager = super.bootstrap();
createMasterRealmAdminUser();
createBaeldungRealm();
return exportImportManager;
}
In the EmbeddedKeycloakConfig we added a couple new bean references/factories (Both found here)
public class EmbeddedKeycloakConfig {
...
#Bean("fixedThreadPool")
public ExecutorService fixedThreadPool() {
return Executors.newFixedThreadPool(5);
}
#Bean
#ConditionalOnMissingBean(name = "springBootPlatform")
protected SimplePlatformProvider springBootPlatform() {
return (SimplePlatformProvider) Platform.getPlatform();
}
In our keycloak-server.json, we removed the default 'concurrenthashmap' name from the mapStorage provider, but still provided the concurrenthashmap properties. This matches other Keycloak 15 examples. As Paul mentioned, the map_storage needs to be enabled, and the simplest solution is to use the system properties -Dkeycloak.profile.feature.map_storage=enabled -Dkeycloak.mapStorage.provider=concurrenthashmap. Similar examples can be found in the Keycloak test suites here and here . HOWEVER: We ran into errors when using these system properties, and decided it is not needed for our code. Luckilly, the 'mapStorage' is no specifically called out in the tutorial; it is only found in the Referenced Tutorial Code. So we simply matched other Keycloak 13+ examples and removed the reference in the keycloak-server.json (shown below) to get rid of the null pointer exception.
//Change this:
"mapStorage": {
"provider": "${keycloak.mapStorage.provider:concurrenthashmap}",
"concurrenthashmap": {
"dir": "${project.build.directory:target}"
}
},
//To This
"mapStorage": {
"provider": "${keycloak.mapStorage.provider:}",
"concurrenthashmap": {
"dir": "${project.build.directory:target}"
}
},
Upgraded the proper infinispan and rest versions with keycloak version. (Tutorial code reference)
< properties>
< keycloak.version>15.0.1</keycloak.version>
< !-- these should be updated together with Keycloak -->
< !-- check keycloak-dependencies-server-all effective pom -->
< infinispan.version>11.0.9.Final</infinispan.version>
< resteasy.version>3.15.1.Final/resteasy.version>
< /properties>
We also added the temp directory location in the SimplePlatformProvider, since this field was required. NOTE: The system temp folder, used in the example, is not recommended for production.
#Override
public File getTmpDirectory() {
return System.getProperty("java.io.tmpdir");
}
I hope someone finds this info useful, Good luck!
I’ve tried the same thing with keycloak 15 and I got the same result. It seems that keycloak’s ConcurrentHashMapStorageProvider isn’t taken in account.
EDIT: after looking into keycloak's code, I've found that MAP_STORAGE functionality is disabled by default (because the feature is tagged as EXPERIMENTAL). To resolve this issue you must launch your springboot app with the following Java option :
-Dkeycloak.profile.feature.map_storage=enabled
Unfortunately, I'm now facing another nullPointerException...
I have applied the changes suggested by #mrieman to the initial baeldung.com tutorial. It works fine with 17.0.1 version of Keycloak as of now. Hope it will help someone. My repository on Github is here

Spring boot 2.4.x cannot handle multi document yml files from config server

Java version: 8
Spring Boot version: 2.4.1
Spring Cloud version: 2020.0.0, specifically I use a Spring Cloud Config Server connected to GIT and our services are Spring Cloud Config Clients.
I have migrated away from using bootstrap.yml and started using spring.config.import and spring.config.activate.on-profile as mentioned in the documentation here and here
My configuration in my service, who is a client to the config server looks like this:
server.port: 9001
spring:
application.name: my-rest-service
config.import: configserver:http://localhost:8888
cloud.config.profile: ${spring.profiles.active}
My configuration in the config server looks like this:
application.yml (has two documents separated by the ---)
logging:
file.name: <omitted>
level:
root: INFO
---
spring:
config.activate.on-profile: dev
logging.level.root: DEBUG
my-rest-sercive.yml (has two documents separated by the ---)
spring:
datasource:
driver-class-name: <omitted>
username: <omitted>
password: <omitted>
---
spring:
config.activate.on-profile: dev
datasource.url: <omitted>
Because there is a profile "dev" active, I successfully get the following 4 configurations from config server:
application.yml: general logging level
application.yml: specific logging for dev
my-rest-sercive.yml: general datasource properties
my-rest-sercive.yml: specific datasource url for dev
I can see these 4 sources successfully being fetched when I use my browser or when I debug or in the logs when I lower the loglevel to trace:
o.s.b.c.config.ConfigDataEnvironment : Adding imported property source 'configserver:https://git.company.com/path.git/file:C:\configservergit\config\my-rest-service.yml'
o.s.b.c.config.ConfigDataEnvironment : Adding imported property source 'configserver:https://git.company.com/path.git/file:C:\configservergit\config\my-rest-service.yml'
o.s.b.c.config.ConfigDataEnvironment : Adding imported property source 'configserver:https://git.company.com/path.git/file:C:\configservergit\config\application.yml'
o.s.b.c.config.ConfigDataEnvironment : Adding imported property source 'configserver:https://git.company.com/path.git/file:C:\configservergit\config\application.yml'
However, notice that because I use multi document yml files, out of these 4 property sources only TWO unique names are used.
In a later step, when Spring creates the data source bean, he complains he cannot find the data source URL. If I debug the spring bean factory I can indeed see that out of the 4 property files returned by the config server, only two have remained (the ones that don't contain the dev profile specific configuration). I assume this is because they have an identical name and they overwrite each other. This is an effect of this piece of code in the MutablePropertySource.class:
public void addLast(PropertySource<?> propertySource) {
synchronized(this.propertySourceList) {
this.removeIfPresent(propertySource); <-- this is the culrprit!
this.propertySourceList.add(propertySource);
}
}
This is a breaking change from Spring 2.3/Spring Cloud Hoxton where it correctly collected all properties. I think spring cloud needs to change the config server so that every document within a yml has has a unique name when returned to Spring. This is exactly how Spring Boot handles multi document yml files, by appending the String (documenyt #1) to the property source name
I found an interesting note about profiles and multi document yml, basically saying it is not supported, but this doesn't apply to my use case because my yml files are not profiles based (there is no -{profileName} in the last part of the file name).
This is a known issue with the new release. We can track the issue here on the spring cloud config server github page.
The workaround seems to be stop using multi document yml files and use multiple distinct files with the profile name in the filename.

Integrate Spring Boot Actuator with New Relic

I am trying to integrate New Relic with Spring Boot actuator. Most of the tutorials and response in StackOverflow itself suggest to use New Relic Java Agent but as per Spring Boot documentation installing Java Agent is not mandatory (unless I misunderstood something) also checked this. So, here is my application.properties currently.
management.metrics.export.newrelic.enabled = true
management.metrics.export.newrelic.api-key = <API_KEY>
management.metrics.export.newrelic.account-id = <ACCOUNT_ID>
logging.level.io.micrometer.newrelic=TRACE
management.metrics.export.newrelic.step=30s
and in the log I am seeing
2021-01-11 12:05:18.315 DEBUG 44635 --- [trics-publisher] i.m.n.NewRelicInsightsApiClientProvider : successfully sent 73 metrics to New Relic.
Based on this logs it looks like it is sending logs. But I have no idea where to see this logs. Ideally I would like to pass app name as well so that I can differentiate metric by app name and preferably by env as well later. Any suggestions?
To add "app name" and "env" to your metrics, you just need to configure the MeterFilter with the common tags:
#Configuration
public class MetricsConfig {
#Bean
public MeterFilter commonTagsMeterFilter(#Value("...") appName, #Value("...") env) {
return MeterFilter.commonTags(Tag.of("app name", appName), Tag.of("env", env);
}
}
Setting the following property you should be able to see what metrics are being sent to NewRelic:
logging.level.io.micrometer.newrelic=TRACE

How to view autoconfigure log output during spring boot server start

How to view autoconfigure log output during spring boot server start
I have created a spring boot application. It uses a shared library (Spring boot jar via maven dependency). Shared library class is loaded via
META-INF/spring.factories
I have mentioned the classes from the library in spring.factories. The job of shared library is to read Vault role id and Vault
secret id value from application.properties and call a REST API and fetch secrets from Vault. After fetching the secret it sets the value again in system property.
for (Map.Entry<String, String> entry : allSecrets.entrySet())
{
System.setProperty(entry.getKey(), entry.getValue());
}
Everything is working as expected. But I am not able to see logs from shared library in my logs.
shared library's package structure is com.myorg.abc. My spring boot package structure is com.myorg.xyz
I tried the following in application properties.
logging.level.root= DEBUG
logging.level.com.myorg.xyz: DEBUG
logging.level.com.myorg.abc: DEBUG
logging.level.org.springframework.boot.autoconfigure.logging=DEBUG
I am able to get logs only from my application but not from shared library. But when I change the shared library Logger.error to System.out, then I am getting the message in my application. How to view shared library's log in my application.
Spring boot initializes logging at least 3 times. The first happens when SpringApplication is loaded. It creates an SLF4J Logger before anything in Spring is accessed. This causes whatever logging implementation you have chosen to initialize. By default, it will use the logging configuration in the Spring jar. With Log4j 2 you can override this by setting log4j.configurationFile to the location of your desired configuration either as a system property or in a log4j.component.properties file.
Everything Spring does will be logged using this configuration until it initializes the logging configuration again, which is controlled by bootstrap.yml. Finally, your application's logging configuration is initialized which is configured either from application.yml or again from bootstrap.yml.
I replaced org.springframework.boot.env.EnvironmentPostProcessor with org.springframework.context.ApplicationListener in Spring.factories and it fixed the issue. I was able to get logs from shared library in invoking application.
Spring.factories
org.springframework.context.ApplicationListener=com.mypackage.MyClassName
MyClassName.java
public class MyClassName implements ApplicationListener<ApplicationPreparedEvent>
{
private static final Logger LOGGER = LoggerFactory.getLogger(MyClassName.class);
#Override
public void onApplicationEvent(ApplicationPreparedEvent applicationPreparedEvent)
{
ConfigurableEnvironment configurableEnvironment = applicationPreparedEvent.getApplicationContext()
.getEnvironment();
String roleId = configurableEnvironment.getProperty(Constants.VAULT_ROLE_ID_LITERAL);
String secretId = configurableEnvironment.getProperty(Constants.VAULT_SECRET_ID_LITERAL);
...
Optional<String> errorMessage = ServiceUtil.validateSystemProperty(roleId, secretId);
if (!errorMessage.isPresent())
{
Map<String, String> secret = ServiceUtil.getSecret(roleId, secretId);
for (Map.Entry<String, String> entry : secret.entrySet())
{
System.setProperty(entry.getKey(), entry.getValue());
}
LOGGER.info("Successfully populated secrets from Vault in system property");
}
else
{
LOGGER.error("Failed to populate secrets from Vault in system property. Error:{}", errorMessage.get());
}
}
}
application.properties
logging.level.com.myorg.abc: DEBUG

Spring Cloud Config client doesn't get values from server

I have been following this tutorial from Baeldung in setting up a Spring Cloud Config server and client. I believe I have set up the server correctly since going to http://localhost:8888/service-config/development/master will seemingly correctly show the json with the user.role defined:
{"name":"service-config","profiles":["development"],"label":"master","version":"d30a6015a6b8cb1925f6e61cee22721f331e5783","state":null,"propertySources":[{"name":"file:///C:.../git/service-config-data/service-config-development.properties","source":{"user.role":"Developer"}},{"name":"file:///C:.../git/service-config-data/service-config.properties","source":{"user.role":"Developer"}}]}
However, I cannot seem to correctly connect the Spring Cloud Config client to the server - the client fails to run with cannot resolve placeholder errors on the #Value("${user.role}") annotation. I've tried adding a default value to the annotation and then manually refreshing the values (using the #refreshscope annotation), but to no avail.
I've looked at the debug and trace logs by running the tools with --debug --trace flags but I cannot find when/if the client is making the call to the server and which config-data .properties file the server is actually looking for. I am a bit lost on how to proceed.
Here is my server application.properties:
spring.application.name=service-config
server.port=8888
spring.cloud.config.server.git.uri=file:///${user.home}/git/service-config-data
spring.cloud.config.server.git.clone-on-start=false
spring.security.user.name=root
spring.security.user.password=toor
And inside the service-config-data folder, there are files with this inside:
user.role=Developer
I have tried files called service-config.properties, service-config-client.properties, service-config-development.properties, and service-config-client-development.properties, and none of them seem to pull through the user.role.
Here is my client:
#SpringBootApplication
#RestController
#RefreshScope
public class ServiceConfigClient {
#Value("${user.role:unknown}")
private String role;
#RequestMapping(
value = "/whoami/{username}",
method = RequestMethod.GET,
produces = MediaType.TEXT_PLAIN_VALUE)
public String whoami(#PathVariable("username") String username) {
return String.format("Hello! You're %s and you'll become a(n) %s...\n", username, role);
}
}
And my client bootstrap.properties:
spring.application.name=service-config-client
spring.profiles.active=development
spring.cloud.config.uri=http://localhost:8888
spring.cloud.config.username=root
spring.cloud.config.password=toor
management.endpoints.web.exposure.include=*
How do I correctly connect the client to the server so that it can successfully pull the config data? Thank you.
My problem was fixed by adding spring.cloud.config.name=service-config to the bootstrap.properties of the service-config-client as well as adding spring-cloud-config-client to and removing spring-cloud-config-server from my pom.xml
There is confusion about client service name and corresponding property sources in config server.
You need to do one of the following:
Change client service name:
//client bootstrap.properties
spring.application.name=service-config
Change folder name in git repo from service-config to service-config-client

Resources