We are using spring cloud config server with a backed git repository to provide properties for a test profile. We have received a requirement to move our keys to vault (hashicorp) and keep regular properties as usual in our properties file.
Before having vault, we were passing the keys through a system property (using -Dxxx=yyy), which was loaded as a regular property source and the app is working as expected.
Now I must have a composite property sources to fetch from the property file and also from vault at the same time. I'm not sure how to pull properties from both vault and git at the same time and offered them to spring cloud config clients.
I've been digging in the documentation and found that we can have composite environment repository, however I cannot make vault and git work at the same time.
I've tried multiple things like putting properties like this:
spring.cloud.config.server.git.uri=file:///E:/Project/git/myappdata
spring.cloud.config.server.vault.host=127.0.0.1
spring.cloud.config.server.vault.port=8200
spring.cloud.config.server.vault.scheme=http
spring.cloud.config.server.vault.backend=secret
spring.cloud.config.server.vault.defaultKey=myapp
Also tested using spring cloud vault and I could fetch the secrets, however they are not provided to my clients.
Have tried to use implement a EnvironmentRepository interface, but this is to create a new repository (and I just want to add 1 vault property to the existing provided repository based on git).
And was going to use the spring event ApplicationEnvironmentPreparedEvent and tried to dynamically append the vault property.
I think I'm overlooking something. Do you know what is the right way to read a vault secret and append it as a property of a regular property file?
It's possible. You should use few profiles for Spring Cloud Config Service.
spring:
profiles:
active: vault, git
cloud:
config:
server:
vault:
order: 1
git:
order: 2
uri: https://some-git-repo.com/
username: user
password: pass
With such config Vault and Git will work together. You will need also to implement support of Vault Token and have it in each configuration client.
With Vault it works a bit differently. It will not get all properties as it does with Git. When client asks for some property with token it will go to Vault and retrieve it. If it's not present it will go and search in git repo. You can specify order in the configuration.
Spring is resolving properties recursively, so you can have property file that will have property placeholder stored in git and served by Config server (application.yml):
database:
password: ${database.secure.password}
and sensitive property stored in Vault, e.g.
vault write secret/clientAppName database.secure.password=SuperSecurePassword
Spring Cloud will automatically resolve your ${database.password} property.
Related
I have a working spring boot application and I am trying to remove some properties from application.yaml file and read them from an embedded config server in the same app. At the moment, I am trying to read properties from file system through "native" profile type, and I am planning to later replace this with S3. Also, I am trying to read the configuration directly from backend repository as explained here https://docs.spring.io/spring-cloud-config/docs/current/reference/html/#_embedding_the_config_server , rather than connecting through an endpoint.
If you want to read the configuration for an application directly from
the backend repository (instead of from the config server), you
basically want an embedded config server with no endpoints. You can
switch off the endpoints entirely by not using the #EnableConfigServer
annotation (set spring.cloud.config.server.bootstrap=true).
I have introduced following changes to my existing application to achieve this.
Added following dependencies to pom.xml
spring-cloud-starter-config
spring-cloud-config-server
spring-cloud-config-client
application-dev.yaml
spring:
could:
bootstrap:
enabled: true
bootstrap.yaml
spring:
profiles:
active: composite
cloud:
config:
server:
composite:
- type: native
search-locations: file:C:\\Users\\chamila\\config-test\\config
# bootstrap: true
The required properties are saved in a different application-dev.yaml file at the above file path. I have not used #EnableConfigServer annotation from my app class, as I want connect directly without the endpoint.
However, still my program is failing to read the properties from the config-server. I tried setting spring.cloud.config.server.bootstrap=true and spring.cloud.bootstrap.enabled=true from both application-dev.yaml and bootstrap.yaml, but it didn't work?
Any idea what I am missing? I note that I never specified how to connect to the config-server in my application-dev.yaml file also. Is that an issue?
Any help is highly appreciated.
I created a sample project of an embedded Configuration Server which uses a custom repository:
https://github.com/taxone/embedded-config-server
The initial problem was that the custom EnvironmentRepository was not available via bootstrap, as described in this issue.
I followed https://cloud.spring.io/spring-cloud-config/reference/html/#_embedding_the_config_server and achieved the expected result.
Add spring-cloud-starter-bootstrap to your dependencies as suggested in https://github.com/taxone/embedded-config-server.
Make sure that composite profile is active when you start the app.
Verify that your native configuration is added by finding a log message from NativeEnvironmentRepository during startup.
(Optional) Remove spring.cloud.bootstrap.enabled from application-dev.yaml.
(Optional) Remove spring-cloud-config-client dependency as it is included in both spring-cloud-starter-config and
spring-cloud-config-server.
(Further steps) Use #RefreshScope on a spring bean to achieve dynamic properties. Requires POST actuator/refresh.
I am using Spring Cloud Vault to store an API Key in production.
From reading the spring.io blog it appears I can use
#Value("${apiKey}")
String apiKey;
to access that key in vault.
This is fine when in production, but is there a way that I can set a default value/ some other way of setting up an apiKey that can be used locally for development? (preferably outside of vault if possible)
As explained in the Spring Boot Reference Guide several sources of configuration properties are consulted. It doesn't really matter where the value for apiKey comes from.
You have at least 3 options:
Set it in the environment,
Pass it as an argument with --apiKey=<your-api-key> when starting the application locally
simply place it in an application.[properties|yaml] used for local development.
Either way the apiKey property will be resolved locally without having to use the vault.
You can set a default value (if the apiKey is not found ) like this :
#Value("${apiKey:MY_KEY_HERE}")
where MY_KEY_HERE is the default value.
The role of the Config Server while using Spring Cloud Config seems to be pretty dumb and hosting it seems to be an unnecessary overhead. Is there a way to get the Clients to directly read configs from the git repo?
You can let Spring Boot look at specific locations for the config files: https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-external-config.html#boot-features-external-config-application-property-files
If you set this location to your cloned git repo (checked out on the branch you need) and solve the update from the origin repository, this might work.
It is achievable via the embedding config server approach
Do refer to Embedding The Config Server section
If you want to read the configuration for an application directly from the backend repository (instead of from the config server), you basically want an embedded config server with no endpoints. You can switch off the endpoints entirely by not using the #EnableConfigServer annotation (set spring.cloud.config.server.bootstrap=true).
Some of my services require API secrets in their configurations. My project is open source, so I cannot store these secrets in the main repository.
Spring Cloud Config has the ability to connect to a private git repository to retrieve the secret configuration, but to do so requires credentials which I, again, cannot store in the main repository.
What is the best practice for storing secrets in an open source application when using Spring Cloud Config?
In both open source and closed source applications, credentials should not be stored with the source code.
Multiple solutions exist to store credentials, you can store them into environment variables, into a property file added in .gitignore or if you want a more elaborate solution you can use a dedicated tool such as HashiCorp Vault. There is an interesting official Spring blog post exploring this solution : Managing Secrets with Vault.
I am using Springboot and developing web services based on those. In my services, I am keeping the configuration at git instead of with project, so that I will be able to change the properties at run-time without restarting the application.
I am using a ConfigServer to fetch the configuartion and all my services are Config Client.
But the issue is my config files are fetched from git based on
-.properties file, so for ex. if my application name is demo and active profile is testing, then the config file which is getting fetched from git is demo-testing.properties.
But the issue I want to have more than one config file. So how should I mention second config file?
Also the second config file is common to other services also?
So actually I want to use same config file for many services which is located at git?