Some of my services require API secrets in their configurations. My project is open source, so I cannot store these secrets in the main repository.
Spring Cloud Config has the ability to connect to a private git repository to retrieve the secret configuration, but to do so requires credentials which I, again, cannot store in the main repository.
What is the best practice for storing secrets in an open source application when using Spring Cloud Config?
In both open source and closed source applications, credentials should not be stored with the source code.
Multiple solutions exist to store credentials, you can store them into environment variables, into a property file added in .gitignore or if you want a more elaborate solution you can use a dedicated tool such as HashiCorp Vault. There is an interesting official Spring blog post exploring this solution : Managing Secrets with Vault.
Related
I am implementing a Spring Boot application which will be providing a multitenant environment. That is achieved in my case by using a database schema for each customer. Example see this project.
Now I am wondering how to implement tenant-specific configurations. I am using #ConfigurationProperties to bundle my property values, but these are getting instantiated once and not for each tenant.
What if I would like to use Spring Cloud Config with multiple tenant specific git repository as an configuration backend. Would it be possible when using a jdbc backend for Spring Cloud Config?
Is there any way with default Spring mechanisms or do I have to implement a database based configuration framework myself?
Edit: For example I have two tenants called Tenant1 and Tenant2. Both are running over the same application in the same context and are writing in the database schemes tenant_1 and tenant_2.
Identification of tenants is happening over keycloak (see Spring Keycloak multi tenant example). So I identify the tenantId from the jwt token and select the database connection like described here.
But now I would need the same mechanism for #Configuration beans. Since #Configuration beans are as far as I know Singletons, so there is always ONE configuration per application scope, and not ONE configuration per tenant.
So using Spring Cloud Config Tenant1 is using https://git-url/tenant1, Tenant2 is using Hashicorp Vault as backend and perhaps Tenant3 will be using a jdbc based configuration backend. And all of that in ONE (of course scalable) application.
In case your application uses tenant specific files (html templates etc), the following can be applied. As I have used the below approach for handling many tenants and works fine and easy to maintain.
I would suggest that you maintain a consistent configuration source (JDBC) for all of your tenant configurations. This helps you have a single source that is cacheable and scalable for your application. Also, you could have your tenants navigate to a configuration page to manage their settings and alter them to suit their needs at any point of time on the fly. (Example Settings: Records Per Page, Theme, Logo, Filters etc...)
Having the tenant configuration in files in git will be a difficult task when you wanted to auto-provision tenant's when they sign-up as it will involve couple of distributed services. Having them in a TenantSettings table with the tenantId as a column could help you get the data in no time and will be easy.
You can use Spring Cloud Config for your scenario and it is adoptable. It is easily configurable and provides out of the box features. For your specific scenario, you can have any number of microservices running yet all controlled by one Spring Cloud Config Server which is connected to one Git Repository. Your all microservices are asking configuration properties from Spring Cloud Config Server and it is directly fetching properties from Git Repository. That repository can have multiple property files. It can hold common properties for all the microservices or specific service based configuration properties. If you want to keep confidential properties more securely, that is also made possible via HashiCorp vault. I will leave an image below for you to get a better idea about this concept.
In the below image, you can see the Git Repository with common configuration property files and specific configuration property files for different services yet in same repository.
I will add another image for you to get a better idea how does this can be arranged with application profiles as well.
Finally I will add something additional to show the power of Spring Cloud Config and out of the box features it allows us to play with. You can automatically refresh configuration properties in running application as well. You can configure Spring Cloud Config to do that. I will add an architectural diagram to achieve that.
References for this answer is taken from Spring in Action, Fifth Edition
Craig Walls
I am using Spring Cloud Config Server first time and have a basic query.
Spring Config server externalises the configuration to a separate git repository.
Why would I create a separate repository just for the configurations?
Is not it advisable to have mono repository with all application code and configurations in a single repo than creating a separate one just for configurations.
We have multiple micro services all present in the same repository. Should not the config server to be one of the micro service present in the same repository where the other application code is?
So, in my multi-module gradle project, I can make config-server as one of the module and give the same repository name as git backed url in config-server. Is this advisable? If yes, where should I keep the configurations in config-server? Inside resources?
Thank you.
When working with microservices it is advisible to have one repository for each microservice. The config server is a microservice as well, therefore it should be put in a separate repository.
Each microservice should have its own independent code repository and your application configuration should never be in the same repository as your source code.
You can read more about this here: Heroku's The Twelve-Factor App. Here you can find 12 best practices to use when building microservices, but for this question I recommend looking at
1st factor: The codebase
3rd factor: The config
I have a spring boot application where I am using some aws services.
The code in openly available in Git.
I don't want to commit AWS secret and access keys which are part of application.properties. I can't add to .gitignore as I want to commit other values of application.properties.
Many are committing to this repo. We are adding these aws keys in local and making sure its not added as part of any commit.
I want to make sure the aws keys in application.properties should not come to git at any cost. Which is the best way to manage these secret keys.?
You shouldn't be placing AWS API keys in application.properties at all. If the application is running on AWS it should be using the IAM role of the server it is running on. If it is not running on AWS it should probably be using environment variables.
Please review the documentation on this subject here.
Thanks to #Mark B. I would prefer using Java system properties as we can maintain them at application level. Env variables will be at system level which is not really needed and it may lead to conflicts.
while running an spring-boot jar with mvn, it can be done as below
mvn spring-boot:run -Dspring-boot.run.jvmArguments="-Daws.accessKeyId=XXXXXXXXXXX -Daws.secretKey=XXXXXXXXXXX"
if running for IDE like Eclipse or IntelliJ, it should be added as VM Options.
-Daws.accessKeyId=XXXXXXXXXXX -Daws.secretKey=XXXXXXXXXXXX
After this AWS client object can be built as usual.
As an example,
SNS client can be build by
AmazonSNS snsClient = AmazonSNSClient.builder().withRegion(Regions.US_EAST_1).build();
SES client can be built by
AmazonSimpleEmailService emailClient = AmazonSimpleEmailServiceClientBuilder.standard().withRegion(Regions.US_EAST_1).build();
The role of the Config Server while using Spring Cloud Config seems to be pretty dumb and hosting it seems to be an unnecessary overhead. Is there a way to get the Clients to directly read configs from the git repo?
You can let Spring Boot look at specific locations for the config files: https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-external-config.html#boot-features-external-config-application-property-files
If you set this location to your cloned git repo (checked out on the branch you need) and solve the update from the origin repository, this might work.
It is achievable via the embedding config server approach
Do refer to Embedding The Config Server section
If you want to read the configuration for an application directly from the backend repository (instead of from the config server), you basically want an embedded config server with no endpoints. You can switch off the endpoints entirely by not using the #EnableConfigServer annotation (set spring.cloud.config.server.bootstrap=true).
We are using spring cloud config server with a backed git repository to provide properties for a test profile. We have received a requirement to move our keys to vault (hashicorp) and keep regular properties as usual in our properties file.
Before having vault, we were passing the keys through a system property (using -Dxxx=yyy), which was loaded as a regular property source and the app is working as expected.
Now I must have a composite property sources to fetch from the property file and also from vault at the same time. I'm not sure how to pull properties from both vault and git at the same time and offered them to spring cloud config clients.
I've been digging in the documentation and found that we can have composite environment repository, however I cannot make vault and git work at the same time.
I've tried multiple things like putting properties like this:
spring.cloud.config.server.git.uri=file:///E:/Project/git/myappdata
spring.cloud.config.server.vault.host=127.0.0.1
spring.cloud.config.server.vault.port=8200
spring.cloud.config.server.vault.scheme=http
spring.cloud.config.server.vault.backend=secret
spring.cloud.config.server.vault.defaultKey=myapp
Also tested using spring cloud vault and I could fetch the secrets, however they are not provided to my clients.
Have tried to use implement a EnvironmentRepository interface, but this is to create a new repository (and I just want to add 1 vault property to the existing provided repository based on git).
And was going to use the spring event ApplicationEnvironmentPreparedEvent and tried to dynamically append the vault property.
I think I'm overlooking something. Do you know what is the right way to read a vault secret and append it as a property of a regular property file?
It's possible. You should use few profiles for Spring Cloud Config Service.
spring:
profiles:
active: vault, git
cloud:
config:
server:
vault:
order: 1
git:
order: 2
uri: https://some-git-repo.com/
username: user
password: pass
With such config Vault and Git will work together. You will need also to implement support of Vault Token and have it in each configuration client.
With Vault it works a bit differently. It will not get all properties as it does with Git. When client asks for some property with token it will go to Vault and retrieve it. If it's not present it will go and search in git repo. You can specify order in the configuration.
Spring is resolving properties recursively, so you can have property file that will have property placeholder stored in git and served by Config server (application.yml):
database:
password: ${database.secure.password}
and sensitive property stored in Vault, e.g.
vault write secret/clientAppName database.secure.password=SuperSecurePassword
Spring Cloud will automatically resolve your ${database.password} property.